What happened
It was reported that on 16 October 2025, Australia’s eSafety Commissioner issued legal notices to four AI companion services, Character.AI, Nomi, Chai and Chub AI, demanding answers about how they were protecting children using their chatbots.
Unlike ordinary chatbots, AI companions are designed to build ongoing relationships with users, remembering conversations and behaving like friends, confidants or romantic partners. The regulator was concerned about reports of children as young as ten spending hours talking to these systems, sometimes encountering sexually explicit conversations or responses involving self-harm and suicide.
When eSafety published its findings in March 2026, it identified serious weaknesses in the protections offered by the four services. None had robust age verification, three did not refer users to crisis or mental health support when self-harm came up, and two had no dedicated trust and safety staff.
The companies responded differently, with Character.AI introducing stronger protections for younger users, Chub AI blocking access from Australia, and others making changes to their services. No fines were imposed as part of the transparency notice process, although Australia’s online safety rules provide for stronger enforcement where legal requirements are breached.
What an auditable version would have shown
The regulator needed to understand whether these companies were genuinely protecting children or simply describing safeguards that sounded reassuring.
An auditable system would have recorded which age checks were applied, what restrictions were active and whether the chatbot attempted to generate responses outside those boundaries.
A ConductRecord could preserve evidence that safety checks were performed, while a PersonaGuard could identify responses that crossed the permitted limits for younger users. This could be done without storing children’s private conversations, reducing the additional privacy risks created by collecting sensitive messages.
Where the gap was
The problem was not simply that children could access AI companions, but that services designed to encourage emotional connection did not consistently have safeguards appropriate for younger users.
Headlights identifies persona and guardrail drift, where a chatbot’s conversational behaviour moves beyond the boundaries it should observe, particularly when the interaction becomes personal or emotionally sensitive.
There is a human behaviour behind this that deserves attention. People naturally respond to something that appears interested in them, remembers what they say and offers constant reassurance, even when they know they are speaking to software.
For children, who are still developing their judgement about relationships and trust, those interactions can be particularly difficult to navigate. A chatbot that feels like a friend may be given more authority than its reliability or understanding warrants.
The eSafety findings concern weaknesses in providers’ protections, rather than establishing that every chatbot or every conversation caused harm.
What governance should have looked like
Companies offering AI companions need to establish who can access their services, what conversations are appropriate and how the system should respond when someone appears vulnerable.
Age checks should be meaningful rather than relying solely on a date entered during registration, while safeguards against harmful content should remain active throughout conversations rather than depending on the chatbot recognising that a user is a child.
A PersonaGuard could enforce restrictions on inappropriate responses, while a ConductRecord could document when those safeguards were triggered and whether the system responded appropriately.
The challenge is that these products are designed to feel personal, which makes the responsibility to establish boundaries even more important. A system built to keep someone talking must also recognise when continuing the conversation is not the safest response.
Failure Pattern: Australia’s eSafety Commissioner identified weaknesses in age assurance and child safety protections across four AI companion services, raising concerns about children’s exposure to inappropriate or harmful conversations.
Governance Principle: AI companions must have enforceable safety boundaries, meaningful age protections and evidence that those safeguards work, particularly when their users may be children.
The reference implementation of PersonaGuard and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
This entry discusses child safety and self-harm. In Australia, Lifeline is available on 13 11 14 and Kids Helpline on 1800 55 1800.
Sources
- eSafety requires providers of AI companion chatbots to explain how they are keeping Aussie kids safe (eSafety Commissioner)
- Findings from transparency notices on AI companion apps: October 2025 (eSafety Commissioner)
- eSafety report shows AI companions are putting children at risk (eSafety Commissioner)
- eSafety Commissioner issues notices to 4 AI companies (MinterEllison)