What happened
It was reported that in May 2023, the National Eating Disorders Association (NEDA) was preparing to replace its human helpline with a chatbot called Tessa, designed to help people at risk of developing eating disorders.
The decision followed a vote by helpline staff to unionise, although NEDA denied any connection between the two events. Tessa had originally been developed as a scripted prevention programme, not as a replacement for trained people supporting those already experiencing eating disorders or seeking urgent help.
Problems emerged when advocates testing the chatbot discovered it was recommending calorie counting, weight monitoring and reducing food intake, advice that could be particularly harmful to someone recovering from an eating disorder.
The chatbot had been given a generative AI capability that allowed it to produce responses beyond its original approved scripts, a change NEDA said had been introduced by its technology provider without its knowledge or approval.
NEDA suspended Tessa on 30 May 2023, but the human helpline still closed as planned shortly afterwards, leaving people without either service for a period.
What an auditable version would have shown
An auditable system would have recorded whether Tessa’s responses came from its approved prevention programme or were generated independently, making it possible to identify when the chatbot began providing advice outside its intended purpose.
It would also have documented when the generative capability was introduced and which responses were affected, allowing NEDA to understand the scale of the problem rather than relying on individual complaints to discover it.
Where the gap was
Headlights identifies scope drift and a rules-behaviour gap, where a tool designed for a limited purpose begins operating beyond the boundaries that originally made it suitable for use.
Tessa was developed to provide structured prevention information, but the addition of generative responses meant it could begin offering advice that had not been reviewed for people experiencing eating disorders.
There is a familiar human tendency behind this failure, where organisations gradually expand the responsibilities of a tool because it appears capable of doing more, without fully considering whether it is appropriate for the people who depend on it.
The problem was not simply that the chatbot gave inappropriate advice, but that a prevention tool was being moved towards a support role it had never been designed to fulfil.
What governance should have looked like
A chatbot supporting vulnerable people should remain within clearly defined limits, particularly when advice that might appear harmless in an ordinary conversation could be dangerous in the context of an eating disorder.
A ConstraintGate could restrict responses to clinically approved material, while a PersonaGuard could prevent the chatbot from moving beyond its prevention role into personalised dietary advice. A ConductRecord could document the source of each response and identify when changes to the system altered its behaviour.
Any changes introduced by a technology provider should also require review and approval before reaching users, with access to trained human support maintained for situations beyond the chatbot’s capabilities.
The lesson is that a system being technically capable of answering a question does not mean it is qualified to answer it, especially when the person asking may be vulnerable.
Failure Pattern: An eating-disorder prevention chatbot began providing potentially harmful dieting advice after gaining the ability to generate responses outside its original scripted programme.
Governance Principle: AI tools used in sensitive health settings must remain within their approved purpose, with controlled changes, clinically reviewed content and access to human support when needed.
The reference implementation of ConstraintGate, PersonaGuard and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- An eating disorders chatbot offered dieting advice, raising fears about AI in health (NPR / KFF Health News)
- Eating Disorder Helpline Fires Staff, Transitions to Chatbot After Unionization (Vice)
- The thinking behind the controversial eating disorder chatbot Tessa (STAT)