What happened
It was reported that in October 2023, New York City Mayor Eric Adams launched MyCity, an AI chatbot designed to help small-business owners understand the rules and regulations affecting their businesses. Built using Microsoft Azure and OpenAI technology, the service was intended to make government information easier to access, allowing people to ask questions about employment, housing, permits and other legal requirements without having to navigate multiple government websites.
The city had invested approximately half a million dollars in developing the service, which the mayor promoted as an example of how artificial intelligence could improve access to government. For small-business owners trying to understand their legal obligations, the appeal was straightforward: instead of searching through legislation and government guidance, they could ask a question and receive an answer in ordinary language.
In March 2024, an investigation by The Markup and The City found that the chatbot was providing incorrect advice about several established laws. When asked whether a landlord could refuse to rent to someone using a Section 8 housing voucher, MyCity indicated that they could, despite New York City’s prohibition on discrimination based on lawful sources of income. It also told users that employers could retain workers’ tips, dismiss employees for complaining about sexual harassment and refuse to accept cash, even though the relevant laws prohibited those actions.
The journalists tested variations of the questions across different sessions and compared the chatbot’s responses with the applicable legislation. They found that MyCity sometimes gave contradictory answers to similar questions, raising concerns about whether business owners could rely on the information it provided. These were not obscure legal issues, but ordinary compliance questions that employers, landlords and retailers might reasonably ask a government service.
Following the investigation, the Adams administration acknowledged that some of the chatbot’s answers were wrong. The mayor said the system was inaccurate in certain areas, while the city updated its website to describe MyCity as a beta product that could provide incomplete or incorrect information. Despite the findings, the chatbot remained available to the public as the city continued working on improvements.
The service continued operating into 2026, when the incoming administration under Mayor Zohran Mamdani announced that it would be discontinued. The decision followed nearly two years of public criticism about the reliability of a government chatbot that had been promoted as a way to help businesses understand their legal responsibilities.
What an auditable version would have shown
According to The Markup’s investigation, the journalists had to test MyCity themselves to establish what advice it was giving businesses. They submitted questions, recorded the answers and compared them with the relevant laws, uncovering incorrect and inconsistent responses that raised concerns about the reliability of the service.
The public reporting does not establish whether New York City retained complete internal records of the chatbot’s conversations or what monitoring arrangements were in place. It does, however, show that the incorrect advice became publicly documented through an external investigation rather than through findings the city had released about the system’s performance.
An auditable record could have preserved the questions users asked, the answers MyCity provided, the model version operating at the time and any government documents used to support those answers. For questions about legal obligations, the record could also have shown whether the response had been checked against an identifiable law or official guidance before being presented to the user.
This would have made it possible to examine more than individual examples of incorrect advice. If the chatbot repeatedly gave different answers to questions about housing discrimination, employment rights or payment requirements, those variations could have been identified and investigated. Regular testing against verified legal information could also have helped establish whether particular subjects were producing more errors than others.
A ConductRecord is designed to preserve evidence of what a system said and the information it relied on, while a VerificationGate could check whether answers about legal requirements were supported by approved sources. Together, these controls could help a public agency identify unsupported advice before it reached users and investigate errors that had already occurred.
Where the gap was
The problem was not simply that MyCity occasionally produced an incorrect answer. The city had introduced the chatbot as a way for small-business owners to understand government requirements, placing it in a position where people might reasonably rely on its responses when deciding how to operate their businesses.
When the system told a landlord that refusing a tenant with a housing voucher was permitted, or suggested that an employer could retain workers’ tips, it was providing information that contradicted existing legal protections. The consequences of following such advice could extend beyond the person asking the question, affecting tenants, employees and customers whose rights were protected by those laws.
Headlights identifies overconfidence in the way the chatbot presented incorrect legal information without making clear that its answers might be unreliable. The system gave responses that appeared to explain established legal requirements, even when those explanations contradicted the law. The problem was made more serious by the fact that the information came through an official government service rather than an independent commercial chatbot.
The incident also reveals a rules-behaviour gap between the laws the city was responsible for administering and the information its own chatbot provided. New York City had established protections concerning housing discrimination, employment rights and cash payments, yet the system intended to help businesses understand those requirements sometimes advised conduct that the law prohibited.
A further governance concern was the city’s decision to keep the service available after the errors became public. The administration acknowledged the problems and added warnings about the chatbot’s limitations, but the service continued operating. A disclaimer could alert users that an answer might be wrong, although it could not establish whether a particular response was legally correct or prevent someone from relying on it.
The incident also demonstrates the importance of independent detection. Journalists identified and documented the incorrect advice by testing the chatbot against existing laws, showing how external scrutiny can reveal problems in systems that organisations have already deployed. The available reporting does not establish whether the city had previously detected the same errors internally, so the investigation should not be treated as proof that no internal monitoring existed.
These behavioural classifications are Headlights’ analysis of the reported conduct, not findings made by a court or regulator.
What governance should have looked like
A government chatbot providing information about legal obligations needs stronger controls than a system answering general enquiries. When a business owner asks whether an action is permitted by law, the answer should be based on current legislation or verified government guidance, rather than on what a language model generates from its training data.
A VerificationGate could check proposed answers against approved legal sources before they are shown to users. Where the system cannot identify a reliable source supporting its response, it should explain that it cannot confirm the answer and direct the person to an appropriate government service or qualified adviser. This would allow the chatbot to assist with straightforward enquiries without presenting unsupported legal interpretations as established facts.
The city could also have tested the system regularly using questions with known legal answers, including the types of questions raised in The Markup’s investigation. Testing should cover different ways of asking the same question, since a system that answers correctly in one conversation may provide a contradictory response in another. Results could then be reviewed by subject area, allowing the city to restrict or suspend answers on topics where the system was repeatedly unreliable.
A ConductRecord could preserve the question, the response, the sources consulted and the verification result for each interaction, subject to appropriate privacy protections. Aggregated information about error rates, unsupported answers and recurring problem areas could help officials assess whether the service was performing reliably and whether continued deployment was justified.
Public reporting of these results would also improve accountability. Residents and businesses should not have to depend entirely on journalists discovering incorrect answers before learning that a government information service has reliability problems. Where a public agency uses AI to explain legal requirements, it should be able to demonstrate how the information is checked, how errors are detected and what happens when the system provides advice that contradicts the law.
Failure Pattern: a government chatbot provided incorrect information about established legal requirements, including housing discrimination, employment protections and cash payments. The errors were documented by journalists, and the service remained available after the city acknowledged problems with its answers.
Governance Principle: AI systems providing information about legal obligations should rely on verified and current sources, be tested regularly against established requirements and refer questions for human assistance when an answer cannot be confirmed. Public agencies remain responsible for the reliability of information delivered through their services.
The reference implementation of VerificationGate and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Malfunctioning NYC AI Chatbot Still Active Despite Widespread Evidence It’s Encouraging Illegal Behavior (The Markup)
- NYC’s Microsoft-Powered Chatbot Tells Business Owners to Break the Law (CX Today)
- Mamdani to kill the NYC AI chatbot we caught telling businesses to break the law (The Markup, January 2026)
- We asked NYC’s new chatbot our real estate questions (Brick Underground)
- NYC’s New Mayor Is Killing the City’s Faulty Chatbot (VICE)