# HD-INC-228: Japan Postponed an AI Tool for Child Abuse Custody Decisions After Doubtful Test Scores

> Japan's Children and Families Agency built an AI tool to score risk in child abuse cases and support decisions on taking children into temporary protective custody. A report for the agency records that in a test about 60 percent of scores raised doubts, including serious oversights in cases veteran caseworkers said needed immediate custody. Release was postponed.

- Source: https://useheadlights.com/library/HD-INC-228/
- Publisher: Headlights (useheadlights.com), author Ellie Harris
- Published: 2026-10-04
- Company: Children and Families Agency (Japan)
- Sector: Child welfare
- Country: Japan
- Year: 2026
- Behaviour in the record (Headlights reading): Not established
- Governance domains: Safety-critical systems, Human oversight
- Licence: CC BY-NC 4.0. Cite as: Harris, E. (2026). Japan Postponed an AI Tool for Child Abuse Custody Decisions After Doubtful Test Scores. Headlights AI Incident Library, HD-INC-228. https://useheadlights.com/library/HD-INC-228/

## What happened

An AI tool designed to help protect children in Japan raised serious concerns during testing. In some cases, its scores failed to reflect the urgency identified by experienced child welfare officers, who believed the children needed to be taken into protective custody immediately.

According to a government report, questions arose about the AI's scores in around 60 percent of test cases.

Japan's Children and Families Agency had been developing the tool since fiscal 2021 to help child guidance centres decide whether children suspected of being abused should be taken into temporary protective custody.

A prototype was completed at the end of fiscal 2023 and tested in fiscal 2024. The findings were documented in a report prepared by PwC Consulting for the agency, dated March 2026 and published on 21 May 2026.

## What an auditable version would have shown

The report explains why the AI and experienced caseworkers sometimes reached different conclusions.

The AI assessed children using a fixed set of questions. According to the report, its scores were correct based on those assessment items. But experienced caseworkers had information that the questions did not cover. When they considered that information, they sometimes reached the opposite conclusion.

The report also found that the project had collected less data than originally expected.

An auditable system would have shown where the AI's scores differed from caseworkers' judgements and what information the assessment had missed.

## Where the gap was

The agency's report says the tool needs further improvements and that its release has been postponed. The agency plans to move towards AI that learns from caseworkers' free-text case records, rather than relying solely on fixed assessment items.

It was reported that concerns had been raised at a government advisory subcommittee in December 2024. Press reports in March 2025 described the decision to put the release on hold and said development had cost about 1 billion yen. These details came from the press, rather than the agency's report.

A VerificationGate is designed to check an output against a trusted reference before it is used, such as an experienced caseworker's judgement on the same case.

A MetricRecord keeps signed results, such as how often the tool's score and the caseworker's judgement disagreed, that the agency, auditors or parliament can check.

VerificationGate and MetricRecord are Headlights designs.

## What governance should have looked like

According to the agency's report, the tool's release was postponed following testing. The sources reviewed do not identify any decision about a child made using the tool.

For AI used to assess whether children need protection, best practice would be to test its scores against experienced caseworkers' judgements before release. That testing should record where the AI misses important risks and examine information that falls outside its assessment questions.

**Failure Pattern:** according to a report for Japan's Children and Families Agency, an AI tool built to score risk for child protective custody produced doubtful scores in about 60 percent of test cases, including serious oversights.

**Governance Principle:** an AI tool that scores risk to children should be tested against experienced caseworkers' judgement before release, and its misses should be recorded.

The reference implementation of VerificationGate and MetricRecord is open source. It lives at [github.com/saffronandindia/headlights-oss](https://github.com/saffronandindia/headlights-oss), Apache 2.0 licensed and free to install. The repository is public now.

## Sources

- [児童相談所業務における AI の技術開発の在り方に関する検討事業 事業報告書 (PwC Consulting for the Children and Families Agency, March 2026, published 21 May 2026)](https://www.cfa.go.jp/assets/contents/node/basic_page/field_ref_resources/d51f146d-2461-4178-893d-87204cc05f8e/87f02578/20260521_policiesjidougyakutaichosakenkyu-r07_01.pdf)

## Related

- [HD-INC-129](/library/HD-INC-129)