What happened
It was reported by Forbes on 2 October 2024 that a father received a Google Alert telling him somebody had created a chatbot on Character.AI using his murdered daughter’s yearbook photograph and name. It was reported that he contacted the company’s support team and received an automated response saying his complaint was being reviewed. A Character.AI spokesperson said the chatbot had been removed for violating its policies on impersonation.
It was then reported by the BBC on 30 October 2024 that the Telegraph had found characters imitating two British teenagers who had died. The BBC reported that the company’s terms of service ban impersonating any person, that the company said it moderated avatars proactively and in response to user reports, and that it deleted the chatbots after being alerted to them.
It was then reported by Futurism on 20 March 2025 that at least four publicly facing characters used the name and likeness of a teenager whose death is the subject of proceedings against the company, reachable through accounts listed as belonging to minors and easily searchable. Fortune reported the same day that lawyers for the teenager’s mother had found several more. The characters were deleted soon after Futurism approached the company, which said they violated its terms and that it was adding to its character blocklist.
The company’s own position, stated on its blog in October 2024, is that it conducts proactive detection and moderation of user created characters. In the three cases above, the reporting records removal following notification from a family member or a journalist, and in none of the sources reviewed did the company state that it had identified any of these characters itself. This entry does not treat that as a denial of proactive moderation, which the company asserts.
The Federal Trade Commission issued study orders under section 6(b) on 11 September 2025 to seven companies including Character Technologies, which is not an enforcement action. Character.AI announced on 29 October 2025 that it would remove open ended chat for users under 18, beginning 24 November. It was reported that Google and Character.AI agreed in principle in January 2026 to settle the lawsuits concerning teenage deaths, with no liability admitted. Proceedings filed by Kentucky and Pennsylvania in 2026 are allegations and have not been determined.
What an auditable version would have shown
In each of the three cases a prohibited character was found by someone outside the company: a father with a search alert on his dead daughter’s name, a newspaper that went looking, and a reporter and a family’s lawyers using the platform’s own search. The company says it also moderates proactively, but no published record shows what that moderation checked. A record written when a character is created, saying what was tested and what followed, would show how many prohibited personas were caught before publication and how many only after a report.
Where the gap was
It was reported that impersonating a person was already prohibited by the terms of service, and that characters using the names and photographs of deceased young people were nonetheless publicly reachable and, in the March 2025 case, searchable from accounts registered as belonging to minors. A PersonaGuard tests a persona against the operator’s declared identity rules at creation and again when it replies. A ConstraintGate runs the prohibition as a check before publication. A ConductRecord preserves what was tested and decided. None of these decides what a platform should allow; they make the prohibition demonstrable.
What governance should have looked like
The company’s position is that it moderates proactively and on user reports. The reporting reviewed here records three occasions on which the notification came from outside. Where users can create personas that others can reach, best practice would be to test a prohibition on impersonating real people before a character becomes reachable, to record the check so the organisation can state how much of its enforcement is proactive, and to have removal requests answered by people rather than automatically. Several of the matters above are live and untested.
Failure Pattern: user created personas prohibited by the platform’s own terms were publicly reachable, and in the three reported cases the company learned that a specific prohibited character existed through a family member, lawyers or journalists rather than through a documented proactive detection event.
Governance Principle: an organisation that lets users create personas should be able to test a new persona against its own prohibitions before it is published, and to show what it checked rather than relying on the people depicted to find it.
The reference implementation of PersonaGuard, ConstraintGate and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Character AI hosted a chatbot of a murdered teenager, her father says (Forbes, 2 October 2024)
- ‘Sickening’ Molly Russell chatbots found on Character.ai (BBC News, 30 October 2024)
- Community safety updates (Character.AI, 22 October 2024)
- Report on characters impersonating a teenager, reachable from accounts registered as minors (Futurism, 20 March 2025)
- Report on further chatbots based on his likeness found by lawyers for his mother (Fortune, 20 March 2025)
- FTC launches inquiry into AI chatbots acting as companions (Federal Trade Commission, 11 September 2025)
- Announcement of the withdrawal of open ended chat for users under 18 (Character.AI, 29 October 2025)
- Update on the rollout of the under 18 changes (Character.AI, 21 November 2025)
- Google and Character.AI negotiate first major settlements in teen chatbot death cases (TechCrunch, 7 January 2026)
- Attorney General Coleman sues Character.AI (Commonwealth of Kentucky, 8 January 2026)