What happened
An AI tool designed to help protect children in Japan raised serious concerns during testing. In some cases, its scores failed to reflect the urgency identified by experienced child welfare officers, who believed the children needed to be taken into protective custody immediately.
According to a government report, questions arose about the AI’s scores in around 60 percent of test cases.
Japan’s Children and Families Agency had been developing the tool since fiscal 2021 to help child guidance centres decide whether children suspected of being abused should be taken into temporary protective custody.
A prototype was completed at the end of fiscal 2023 and tested in fiscal 2024. The findings were documented in a report prepared by PwC Consulting for the agency, dated March 2026 and published on 21 May 2026.
What an auditable version would have shown
The report explains why the AI and experienced caseworkers sometimes reached different conclusions.
The AI assessed children using a fixed set of questions. According to the report, its scores were correct based on those assessment items. But experienced caseworkers had information that the questions did not cover. When they considered that information, they sometimes reached the opposite conclusion.
The report also found that the project had collected less data than originally expected.
An auditable system would have shown where the AI’s scores differed from caseworkers’ judgements and what information the assessment had missed.
Where the gap was
The agency’s report says the tool needs further improvements and that its release has been postponed. The agency plans to move towards AI that learns from caseworkers’ free-text case records, rather than relying solely on fixed assessment items.
It was reported that concerns had been raised at a government advisory subcommittee in December 2024. Press reports in March 2025 described the decision to put the release on hold and said development had cost about 1 billion yen. These details came from the press, rather than the agency’s report.
A VerificationGate is designed to check an output against a trusted reference before it is used, such as an experienced caseworker’s judgement on the same case.
A MetricRecord keeps signed results, such as how often the tool’s score and the caseworker’s judgement disagreed, that the agency, auditors or parliament can check.
VerificationGate and MetricRecord are Headlights designs.
What governance should have looked like
According to the agency’s report, the tool’s release was postponed following testing. The sources reviewed do not identify any decision about a child made using the tool.
For AI used to assess whether children need protection, best practice would be to test its scores against experienced caseworkers’ judgements before release. That testing should record where the AI misses important risks and examine information that falls outside its assessment questions.
Failure Pattern: according to a report for Japan’s Children and Families Agency, an AI tool built to score risk for child protective custody produced doubtful scores in about 60 percent of test cases, including serious oversights.
Governance Principle: an AI tool that scores risk to children should be tested against experienced caseworkers’ judgement before release, and its misses should be recorded.
The reference implementation of VerificationGate and MetricRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.