The Markup, now a part of CalMatters, uses investigative reporting, data analysis, and software engineering to challenge technology to serve the public good.
asserted
Markup → use → good
Sign up for Klaxon, a newsletter that delivers our stories and tools directly to your inbox.
asserted
that → sign → inbox
This article is co-reported with THE CITY, a non-profit newsroom that serves the people of New York.
asserted
that → co → York
In a press conference this week on New York City’s $12 billion budget gap, Mayor Zohran Mamdani zeroed in on the previous administration’s artificial intelligence chatbot as one of “a number of different things we’re going to pursue for savings.”
asserted
we → zero → savings
The chatbot, which was released by the Eric Adams administration in fall of 2023, was meant to provide business owners with an accessible way to check city rules and regulations.
asserted
which → release → rules
But as first documented by The Markup and THE CITY, the bot provided answers that, if followed, would lead to illegal behavior by businesses, like taking a cut of employees’ tips.
asserted
that → document → tips
A spokesperson for the mayor, Dora Pekec, confirmed in a text message that the new administration plans to take down the chatbot.
asserted
administration → confirm → chatbot
She said a member of the Mamdani transition team had seen reporting on the bot from The Markup and THE CITY and presented it to the mayor as a possible place to save funds.
asserted
member → say → funds
Artificial Intelligence
NYC’s AI Chatbot Tells Businesses to Break the Law
The Microsoft-powered bot says bosses can take workers’ tips and that landlords can discriminate based on source of income
asserted
landlords → tell → income
At the press conference, Mamdani blamed Adams for the budget shortfall, saying he had been handed “a poisoned chalice.”
asserted
he → blame → chalice
To close the deficit, he said he would raise taxes on the wealthy and corporations and look “under the hood” of the city’s budget for potential savings.
asserted
he → close → savings
When pressed by reporters on what he might cut, he singled out the chatbot.
uncertain
he → press → chatbot
“The previous administration had an AI chatbot that was functionally unusable,” Mamdani said.
asserted
Mamdani → have → chatbot
“It was costing the administration around half a million dollars.
asserted
It → cost → dollars
That, in and of itself, is not something that can bridge this kind of a gap, but it’s an indication of the ways in which money has been spent while refusing to account for the actual costs of what these programs are.”
The bot, built using Microsoft’s cloud computing platform, was part of an ambitious overhaul of digital services in New York called MyCity.
asserted
bot → bridge → York
The project was meant to streamline access to government but was criticized for relying on outside contractors.
asserted
project → mean → contractors
It wasn’t clear how much it cost to maintain the chatbot.
asserted
it → cost → chatbot
Just building the bot’s foundations reportedly cost nearly $600,000, close to the figure Mamdani provided.
uncertain
Mamdani → build → figure
Pekec said they didn’t yet have a date for taking down the bot.
asserted
they → say → bot
Testing by The Markup and THE CITY in 2024 showed that, despite promises from the Adams administration, the chatbot would confidently provide incorrect and potentially harmful information to visitors, even on high-stakes topics.
When asked about housing policy, for example, the bot suggested landlords could discriminate against tenants with Section 8 vouchers.
uncertain
landlords → show → vouchers
Despite being an intended resource for business owners, the bot didn’t know the minimum wage, and told users it was fine to refuse to accept cash for payment despite a city law to the contrary, enacted in 2020.
asserted
it → intend → 2020
After The Markup and THE CITY’s initial report was published, readers continued peppering the bot with sometimes farcical questions, which it continued failing to answer properly.
asserted
it → publish → which
The Adams administration defended the bot, saying it would improve over time.
asserted
it → defend → time
“We’re identifying what the problems are, we’re gonna fix them, and we’re going to have the best chatbot system on the globe,” Adams said at a press conference.
asserted
Adams → identify → conference
“People are going to come and watch what we’re doing in New York City.”
asserted
we → go → City
City administrators soon added disclaimers to the bot advising users to “not use its responses as legal or professional advice.”
asserted
administrators → add → advice
They also improved some of the bot’s answers, but also appeared to limit the kinds of questions the tool was willing to answer.
asserted
tool → improve → questions
Today, the bot advises visitors to “ask an NYC government question only” and cautions that “responses may occasionally produce inaccurate or incomplete content.”
uncertain
responses → advise → content
Visitors must agree to accept the bot’s limitations before using it.
asserted
Visitors → agree → it
The city has taken down the chatbot, writing on the bot’s web page that its “beta test has ended” and directing visitors to NYC.gov for government information.
asserted
test → take → information
The Aporia
