What happened
The National Eating Disorders Association ran a helpline that people in crisis could call or message and reach a trained human. In the spring of 2023, shortly after the helpline’s four workers voted to unionise, NEDA announced, on 31 March, that it would wind the helpline down and direct people instead to Tessa, a chatbot, with the transition expected around the start of June. NEDA denied that the closure was connected to the union.
Tessa was not a general-purpose chatbot. It was a rule-based, scripted eating-disorder prevention programme, designed by the Washington University researcher Dr. Ellen Fitzsimmons-Craft and operated by the digital-health company Cass (formerly X2AI). It had run on NEDA’s site since 2022, and, as its developer later stressed, it was built to help people at risk who had not yet developed an eating disorder, never to replace a helpline or to support people in crisis. Casting a prevention tool in the helpline’s higher-acuity role was already a stretch of what it was designed to do.
There had been a warning months earlier. In October 2022, Monika Ostroff, executive director of the Multi-Service Eating Disorders Association, gave NEDA screenshots of Tessa advising users to avoid “unhealthy” foods and choose “healthy” snacks like fruit, diet-culture language with no place in eating-disorder support. That wording was removed, and Cass said it had been pre-scripted rather than generated. The mechanism mattered: it was a content problem inside the script, fixed by editing the script.
The second failure came by a different route. By 2023 a generative capability had been added that let Tessa compose its own replies rather than stay inside its programme. Advocates who tested it, among them Sharon Maxwell, who lives in recovery, reported that when asked for help the bot offered advice that runs directly against eating-disorder care: counting calories, aiming for a calorie deficit, weighing regularly, and measuring body fat with calipers. For most people that advice is unremarkable; for someone with an eating disorder it is precisely the thinking treatment works to undo. NEDA took Tessa offline around 30 May 2023, days before it was due to replace the helpline, saying the program “may have given information that was harmful and unrelated to the program” it was meant to deliver, and that the generative feature had been added by Cass without NEDA’s knowledge or approval. Fitzsimmons-Craft said the bot had been giving AI-generated “ad-libs” that were never part of its development. The timing left a gap: Tessa was suspended on 30 May, the human helpline staff were let go on 1 June as planned, and for a period NEDA had no live support resource at all, neither the people it had let go nor the tool meant to replace them.
What an auditable version would have shown
When a health-adjacent assistant gives harmful advice, the organisation needs to answer a narrow question quickly: how far outside its approved content did it go, to how many people, and from what point. NEDA learned of both failures because individual advocates tested the bot and spoke publicly. What it could not easily establish was the boundary itself, how often Tessa had left its sanctioned prevention programme and generated unscripted advice, and to whom, because nothing was systematically recording where the bot’s answers came from. An auditable version would record, for each response, whether the answer was drawn from the approved programme content or generated freely, the model or feature version in force, and the output, signed at the time. That record makes the boundary legible, lets the organisation say with evidence how many conversations crossed from scripted support into generated advice, and fixes responsibility precisely, the original scripted bot or the added generative feature, instead of leaving the two sides to attribute it to each other.
Where the gap was
The underlying gap was that a tool serving a vulnerable population was allowed to generate free-form advice in a domain where only approved, clinically grounded content is safe, with nothing confining it to that content and nothing recording when it strayed, and with a vendor able to add that capability without the deploying organisation even knowing. A prevention programme that answers only from vetted scripts is a defined, bounded thing; bolting on a feature that lets it improvise turns it into an open-ended advice-giver without the safeguards that role demands. A ConstraintGate is the control for this: in a sensitive category like eating-disorder support, the assistant is confined to approved, vetted responses, and anything outside that scope is refused or routed to a human rather than generated. A PersonaGuard keeps the tool inside its defined role as a prevention guide, so it does not drift into clinical advice it was never meant to give. A ConductRecord preserves each response and its source, so the organisation can measure adherence to the boundary instead of discovering breaches through its users.
What governance should have looked like
The lesson of Tessa is not that support cannot be assisted by software; it is that in a domain where the wrong sentence is dangerous, the assistant must be confined to content that has been checked, a person must remain reachable, and a tool scoped for prevention must not be quietly promoted into a crisis role it was never built for. Replacing a staffed helpline with a generative chatbot removed both the human fallback and the guarantee that every answer had been vetted, and it did so for exactly the people least able to absorb a harmful response.
The sharper lesson is about the vendor relationship. The harmful advice came from a feature a third party added to a component touching a vulnerable population, and the deploying organisation says it did not know. A safe deployment keeps the bot inside approved material, refuses or escalates anything beyond it, records where each answer came from, and binds the vendor by contract and by control so that no unilateral feature change can alter what the tool says to people in distress. Those safeguards are ordinary in a mature governance framework, and far less costly than a helpline withdrawn at the moment people relied on it.
The reference implementation of ConstraintGate, PersonaGuard, and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- An eating disorders chatbot offered dieting advice, raising fears about AI in health (NPR / KFF Health News)
- Eating Disorder Helpline Fires Staff, Transitions to Chatbot After Unionization (Vice)
- The thinking behind the controversial eating disorder chatbot Tessa (STAT)