What happened
On 16 October 2025, Australia’s eSafety Commissioner, Julie Inman Grant, issued legal transparency notices to four providers of AI companion chatbots, Character.AI, Nomi, Chai, and Chub AI, requiring them to explain how they were keeping children safe. The Commissioner cited reports of children as young as ten spending up to five hours a day with these companions, at times in sexual conversation, and chatbots capable of engaging minors in sexually explicit exchanges and, in some accounts, of encouraging suicidal ideation, self-harm, and disordered eating.
AI companion apps are built to form an ongoing, intimate relationship with the user. The chatbot adopts a persona, remembers earlier conversations, and is designed to feel like a friend, a confidant, or a romantic partner. By 2025 several of these services had large user bases that included children. The notices were issued under Australia’s Basic Online Safety Expectations and covered the reporting period from 1 July to 30 September 2025, requiring the providers to show, not assert, how they were meeting them.
eSafety published its findings in a report in early 2026. None of the four providers, it found, had robust age verification: they relied on app-store age ratings or a user’s own declaration at sign-up, neither of which establishes how old the person actually is. Several did not route signs of self-harm in a user’s messages to crisis or mental-health support; on eSafety’s account some did not warn users that prompting for child sexual abuse material is a crime or report such material to authorities; and two of the services had no staff dedicated to trust and safety at all. In the period after the notices the providers moved at different speeds: Chub AI geo-blocked its service from Australia rather than continue under the requirements; Character.AI introduced age-assurance measures for Australian users in early 2026 and removed the chat function from its under-18 experience; Chai placed its companions behind a paid subscription; and Nomi committed to further age-assurance work. None was fined; the leverage was the notice itself and the obligation to answer it. Separately, the Age-Restricted Material Codes are now law and require companion chatbots either to prevent the generation of age-inappropriate material such as sexually explicit content or to apply appropriate age assurance, and to surface crisis and mental-health information, backed by civil penalties of up to 49.5 million dollars for breaching a compliance direction, a backstop none of the four has yet had to test.
The failure here is not a single wrong answer. It is that a system explicitly designed to inhabit a persona and sustain an intimate relationship had no reliable boundary on who it was talking to or what that persona was permitted to say to them.
What an auditable version would have shown
The regulator’s instrument was a notice asking the providers to explain themselves. Much of what eSafety and the public know about these systems comes from the companies’ own descriptions and from individual reports, because the conversations themselves are private and the providers hold whatever records exist.
An auditable companion service would be able to demonstrate, from signed records rather than assurances, the things the notices asked about: what age-assurance check was applied to a given account, what content boundaries were in force for a user the system had reason to believe was a minor, and when the persona stepped outside its defined scope. The point is not to log private conversations for inspection, which would create its own harm, but to make the safety controls themselves provable. A record for a session would carry fields like the age band applied at the start, for example under-18, the content policy in force for that band, and a flag each time the system blocked an out-of-scope reply or routed a self-harm signal to a crisis response, none of which requires storing the words that were exchanged.
Where the gap was
The gap was in the boundary on the persona, and in the absence of any record that the boundary held.
A PersonaGuard sits between the model and the user and asks whether a given reply is consistent with the agent’s defined identity and permitted scope for this user. For a companion service, the defined scope for an account believed to belong to a child should exclude sexual content and should route any sign of self-harm to a crisis response rather than into the conversation. A reply that crosses that boundary is the event the guard exists to catch and to deny, regardless of which part of the system generated it. A ConductRecord then captures that the check ran and what it decided, so the provider can show, and the regulator can confirm, that the safeguard was real and not nominal.
These services had the opposite design. The persona was the product, and its willingness to go wherever the conversation led was a feature, with no enforced boundary tied to the age of the person on the other end.
What governance should have looked like
A service that markets an intimate AI relationship to a general audience that includes children needs the safety boundary to be a property of the system that it can prove, not a policy it can describe.
Age assurance has to be a real signal that gates what the persona may do, not a checkbox at sign-up. But it is worth being honest about what age assurance can and cannot do: it can check who opened an account, not who is holding the phone. A child who picks up a parent’s unlocked device, already signed in to an adult account, defeats any check made at sign-up. That is why the content boundary matters at least as much as the age signal. A system that will not generate sexual content or follow a user into self-harm, by default and for every account, does not depend on correctly guessing the age of the person typing; it is also why eSafety’s codes let a provider comply either by preventing the generation of such material or by applying age assurance. The persona’s permitted scope has to be enforced on every reply, with sexual content and self-harm content treated as boundaries rather than as topics the model is free to follow. And the operation of those safeguards has to leave a record, so that when a regulator asks how children are being kept safe, the answer is evidence rather than narrative. The provider that can produce that record can stay and answer the notice. The provider that cannot is left with the choice one of these four made, which was to leave.
This entry concerns self-harm and child safety. If you or someone you know needs support in Australia, Lifeline is available on 13 11 14 and Kids Helpline on 1800 55 1800.
The reference implementation of PersonaGuard and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. Anyone can read every line and verify the signatures. The repository is public now.
Sources
- eSafety requires providers of AI companion chatbots to explain how they are keeping Aussie kids safe (eSafety Commissioner)
- Findings from transparency notices on AI companion apps: October 2025 (eSafety Commissioner)
- eSafety report shows AI companions are putting children at risk (eSafety Commissioner)
- eSafety Commissioner issues notices to 4 AI companies (MinterEllison)