# HD-INC-168: Google's Gemma model produced a fabricated criminal allegation about a sitting US senator, and Google withdrew the model from AI Studio while keeping it available through the API > A sitting US senator wrote to Google's chief executive on 30 October 2025, her office publishing the letter the following day, recording that when asked whether she had been accused of rape the Gemma model returned a narrative about a state senate campaign and a state trooper, supported by links the letter says led to error pages and unrelated news articles. Her letter states there was never any such accusation, no such individual and no such news stories, and that the campaign year in the output was wrong. She had raised a similar output about an activist at a Senate Commerce hearing on 29 October 2025, where Google's witness said it is well known that language models hallucinate. Google posted on 31 October that Gemma was built for developers and researchers, was never intended as a consumer tool, and was being removed from AI Studio while remaining available through the API. The activist sued Google in Delaware on 22 October 2025; the court denied Google's motion to dismiss on 24 July 2026 and expressly made no finding on the truth of any allegation. - Source: https://useheadlights.com/library/HD-INC-168/ - Publisher: Headlights (useheadlights.com), author Ellie Harris - Published: 2026-08-25 - Company: Google - Sector: Consumer software - Country: United States - Year: 2025 - Behaviour in the record (Headlights reading): Overconfidence; also Defensive denial - Governance domains: Algorithmic accountability, Human oversight - Licence: CC BY-NC 4.0. Cite as: Harris, E. (2026). Google's Gemma model produced a fabricated criminal allegation about a sitting US senator, and Google withdrew the model from AI Studio while keeping it available through the API. Headlights AI Incident Library, HD-INC-168. https://useheadlights.com/library/HD-INC-168/ ## What happened It was reported in October 2025 that a sitting US senator had asked Google's Gemma model whether she had been accused of rape, and that the model returned a detailed account of an incident during a state election campaign, supported by links styled as news reports. Her office published the letter it sent to Google, stating that there had never been such an accusation, that no such person existed, that the campaign year was wrong, and that the links led to error pages and unrelated articles. Google said Gemma was built for developers and researchers, was not meant for factual questions or for consumers, and removed it from its AI Studio interface while keeping it available through the programming interface. The senator replied that removing it from one interface does nothing to stop developers deploying it elsewhere. Separately, an activist who had found similar output about himself sued Google, and in July 2026 a court refused to dismiss the case while stating it was making no finding about the truth of the allegations. ## What an auditable version would have shown The people the output was about were the ones who discovered the problem. One ran the prompt in his own name and sued; the other's office ran the prompt and published the result. The sources reviewed did not show how often the model produced claims like this, or what checks Google ran before returning them. A record for each answer could show the sources the model used, whether those sources actually supported the claim, and whether the citations were checked before the answer appeared. That would give Google its own record of what happened, rather than leaving the person named in the output to discover it by running the prompt themselves. ## Where the gap was The senator's letter states that the output presented a false factual claim about her and attached links styled as news reports that led to error pages and unrelated articles. A VerificationGate is designed to route a factual claim about a named person to a trusted source for checking rather than relying on model generation alone. A CitationVerifier is designed to test whether a citation supports the claim attached to it before an answer is delivered, and to record whether the check passed. A ConductRecord preserves the prompt, the retrieved documents, the checks and the output, which allows a disputed answer to be traced afterwards rather than reproduced by the person it concerns. None of these decides what a model should be permitted to say, and none resolves the questions before the Delaware court. ## What governance should have looked like Google's stated position is that Gemma was built for developers and researchers and was not meant for factual assistance or for consumers. The senator argued that removing Gemma from one interface did not stop developers using it elsewhere. The sources reviewed do not settle that disagreement, and the proceedings are still unresolved. Where a model is answering factual questions about real people, best practice would be to check those claims against a trusted source before showing them. Citations should also be checked to make sure they actually support what the model is saying, with a record kept of what was checked and what the checks found. **Failure Pattern:** a model made a false claim about a real person and backed it up with links that looked credible but did not support what it said. The people named in the output found the problem, not the operator. **Governance Principle:** when a model makes factual claims about real people, those claims should be checked against a trusted source. If it provides a citation, the organisation should also be able to show that the citation actually supports the claim. The reference implementation of VerificationGate, CitationVerifier and ConductRecord is open source. It lives at [github.com/saffronandindia/headlights-oss](https://github.com/saffronandindia/headlights-oss), Apache 2.0 licensed and free to install. The repository is public now. ## Sources - [Letter to Sundar Pichai, 30 October 2025, carrying the prompt and the output (Office of Senator Marsha Blackburn)](https://www.blackburn.senate.gov/services/files/5651166C-7B30-4BA0-9E86-F3DD9521CA00) - [Blackburn demands answers from Google after Gemma manufactured fake criminal allegations against her (Office of Senator Marsha Blackburn, 31 October 2025)](https://www.blackburn.senate.gov/2025/10/technology/blackburn-demands-answers-from-google-after-gemma-manufactured-fake-criminal-allegations-against-her) - [Blackburn holds Google's feet to the fire after the company dodges accountability (Office of Senator Marsha Blackburn, 21 November 2025)](https://www.blackburn.senate.gov/2025/11/technology/blackburn-holds-google-s-feet-to-the-fire-after-company-dodges-accountability-for-gemma-scandal) - [Transcript of the Senate Commerce hearing of 29 October 2025 (Tech Policy Press)](https://www.techpolicy.press/transcript-senate-hearing-on-jawboning-with-meta-and-google-execs/) - [Google pulls Gemma from AI Studio, carrying Google's statement in full (TechCrunch, 2 November 2025)](https://techcrunch.com/2025/11/02/google-pulls-gemma-from-ai-studio-after-senator-blackburn-accuses-model-of-defamation/) - [Second report carrying the same Google statement (The Register, 3 November 2025)](https://www.theregister.com/2025/11/03/google_pulls_gemma_from_ai_studio/) - [Starbuck v Google LLC, complaint filed 22 October 2025, Superior Court of Delaware](https://www.dhillonlaw.com/wp-content/uploads/2025/10/20251022_Stamped-Complaint-Summons-Praecipe-CIS.pdf) - [Starbuck v Google LLC, opinion of 24 July 2026 denying the motion to dismiss](https://storage.courtlistener.com/pdf/2026/07/24/starbuck_v._google_llc.pdf) ## Related - [HD-INC-057](/library/HD-INC-057) - [HD-INC-163](/library/HD-INC-163) - [HD-INC-034](/library/HD-INC-034)