What happened
In February 2024 Google’s Gemini app, the rebranded successor to Bard, could generate images from a text prompt. Within days users began posting what it returned when asked for people in historical settings. Asked for a 1943 German soldier, it produced a racially diverse set of figures in Wehrmacht uniform. Asked for America’s founding fathers, it returned a Black man among them. It generated Black Vikings, a female pope, and, in some cases, refused prompts that asked for white people while complying with the equivalent prompt for other groups.
The images were not the model misunderstanding the request. They were the product of an instruction Google had added behind the scenes. To stop the model defaulting to images of white people, a known failure mode of image generators trained on skewed data, Google had tuned Gemini to show a range of people and to resist prompts that would return only one group. That instruction was applied to prompts without regard to whether the context made a diverse result appropriate. A prompt for “a nurse” and a prompt for “a 1943 German soldier” were treated the same way. The first benefits from variety; the second has a specific historical referent that the injected variety contradicts.
A second failure compounded the first. Over time the model had become more cautious than intended and began refusing anodyne prompts it wrongly read as sensitive. The combined effect, historically false images on one hand and blanket refusals on the other, read to many as the system imposing a worldview rather than answering the question.
On 22 February 2024 Google paused Gemini’s image generation of people. Prabhakar Raghavan, then a senior vice president, published a post, “Gemini image generation got it wrong. We’ll do better,” setting out the two faults: the tuning to show a range of people had not accounted for cases that should clearly not show a range, and the model had grown over-cautious and refused too much. Chief executive Sundar Pichai told staff in an internal note that the responses had been “completely unacceptable.” The episode fed a wider argument about bias in the product and weighed on Alphabet’s share price. On 28 August 2024 Google restored the generation of images of people through a new model, Imagen 3, with added safeguards, initially declining photorealistic images of identifiable people and images of children.
What an auditable version would have shown
The failure was diagnosed the way these failures usually are: members of the public screenshotting outputs and posting them. What Google did not show, image by image, was the step that produced them. The user had not asked for a diverse 1943 German soldier; a hidden modification to the prompt had. Nothing in the output distinguished what the user requested from what the system silently added.
An auditable version records, for each generation, the user’s prompt, any system-level modification applied to it, the model version in force, and the image produced. With that record, “an ethnically diverse Wehrmacht” is visible as the result of a prompt rewrite that fired where it should not have, not as a quirk of the model, and the question becomes measurable rather than anecdotal: how often did the diversity modification fire on prompts with a specific historical or factual context, and how often did the refusal guardrail trip on harmless requests. That is the evidence that would have surfaced the problem in testing, before launch, instead of in a viral thread.
Where the gap was
The model was not the failure; it did what the surrounding system asked of it. The failure sat one layer up: a safety control built for a single context was given global scope, applied to every prompt regardless of whether that prompt warranted it. Gemini could render a historically appropriate image when the constraint did not override it. The correction for one bias, the model’s tendency to default to white people, was applied as a single blanket rule to every prompt, with nothing checking whether the context made it appropriate.
A ConstraintGate is the control. A constraint on what a generative system produces should be conditional on context, not uniform: a default toward variety is reasonable for an open prompt like “a doctor,” and wrong for a prompt with a specific referent like “a 1943 German soldier.” A gate that distinguishes the two applies the modification only where it fits and leaves the historically specific prompt alone. The second failure, refusals firing on anodyne prompts, is the same shape: a guardrail acting without regard to whether the particular request warranted it. A ConductRecord of the modification applied to each generation makes the behaviour testable, so an organisation can measure how its own guardrails fire across the range of prompts before the public does it for them.
What governance should have looked like
The lesson is not that a model maker should ignore the bias in image training data. It is real, and correcting it is legitimate. The lesson is that a correction applied uniformly to every output, with no check on whether the context makes it appropriate and no record of what it changed, trades one visible failure for another. A constraint on a generative system should be conditional, applied where it fits and withheld where it does not, and it should leave a record of what it altered and why, so the behaviour can be checked against the full range of prompts before launch rather than discovered in screenshots after it.
Google had the model quality to get this right. What it lacked, in the first weeks, was a context-aware gate between the prompt and the generator, and a record of the modification it was silently applying to every image of a person.
The pattern is worth naming, because it is not Google’s alone. A safeguard built for one context becomes a failure when it is applied, silently, to every context. The control has to be conditional rather than universal: scoped to where it fits, withheld where it does not, and recorded either way. The model was not the problem. The scope of the control was.
The reference implementation of ConstraintGate and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Gemini image generation got it wrong. We’ll do better. (Google blog, Prabhakar Raghavan)
- Google suspends Gemini AI image generation of people after historical inaccuracies, including racially diverse Nazi soldiers (Variety)
- Google to pause Gemini AI model’s image generation (CNN Business)
- Google says it’s fixed Gemini’s people-generating feature (TechCrunch, August 2024)