What happened
From 2014 Amazon built an experimental tool to automate the top of recruiting: read a résumé, return a score of one to five stars, surface the best candidates. The model was trained on about ten years of résumés the company had already received. Because the applicant pool in technical roles had been mostly male, the training data was mostly male, and the model learned from it that male candidates were the pattern to favour.
The result was discrimination the system taught itself. It marked down résumés that contained the word “women’s,” as in “women’s chess club captain,” and reportedly downgraded graduates of two women’s colleges. Engineers edited the model to be neutral on those specific terms, but could not be confident it had not found other, subtler proxies for gender. Amazon lost confidence that the tool was neutral across the board and abandoned the project by 2017. The story became public when Reuters reported it in October 2018.
Amazon has said the tool was never used by its recruiters to evaluate candidates. Reuters reported that recruiters did look at the tool’s recommendations, but never relied on them alone. Even on Amazon’s account, the episode is the clearest early demonstration of a now-familiar pattern: a model trained on a biased history will reproduce that bias, confidently and at scale, unless something is built to catch it.
What an auditable version would have shown
The question a regulator or a rejected applicant asks is narrow: did the system treat a protected group differently, by how much, and from what point. Amazon caught the “women’s” penalty internally, but could not establish that the model was clean elsewhere, because nothing was systematically measuring outcomes by group as the model scored. An auditable version records each scoring decision and the features behind it, and computes a standing metric of selection rates across groups, bound to those records, so disparate impact shows up as a number that can be recomputed and checked rather than as a discovery an engineer happens to make. That record turns “we think it is biased” into “here is the disparity, measured.”
Where the gap was
The gap was that a model was scoring people with no continuous measurement of whether it scored protected groups differently, and no record tying each decision to the features it rested on. Editing out the word “women’s” treats the symptom; the disease is proxy bias the model can rediscover through other correlates of gender. The control is a MetricRecord: a signed, recomputable measure of selection rates across groups, computed over the scoring records, so disparate impact is monitored rather than stumbled upon. A ConductRecord preserves each decision and its inputs, which is what makes the metric trustworthy and the individual decision explainable. A ConstraintGate can bar the use of features that act as protected-class proxies.
What governance should have looked like
Amazon’s own conclusion was the right one: a tool that cannot be shown to be fair should not decide who gets seen by a recruiter. The lesson for everyone who did not scrap their tool is that fairness is not a property you assert once at launch. It is a number you measure continuously, against the records of what the system actually did, because a model trained on the past will carry the past’s bias forward by default. The control is not a one-time audit. It is a standing measure, bound to the decision record, that makes disparity visible while it is still a score and not yet a lawsuit.
The reference implementation of MetricRecord, ConductRecord, and ConstraintGate is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Amazon scraps secret AI recruiting tool that showed bias against women (Reuters)
- Amazon ditched AI recruitment software because it was biased against women (MIT Technology Review)
- Why Amazon’s Automated Hiring Tool Discriminated Against Women (ACLU)