What happened
It was reported that in November 2019 the software developer David Heinemeier Hansson wrote that Apple Card had offered him roughly twenty times the credit limit it offered his wife, even though the couple filed jointly and she had the higher credit score. The post spread quickly, and Apple’s co-founder Steve Wozniak added that he had seen a similar gap of about ten times between his own limit and his wife’s. The card was a Goldman Sachs product, underwritten by the bank, and the accounts of customers and even of Apple staff, who reportedly could not explain the individual decisions when asked, turned a pair of tweets into a public question about whether the algorithm behind the card discriminated against women.
The New York State Department of Financial Services opened an investigation into Goldman Sachs Bank USA. Its report, released on 23 March 2021, described a review of several thousand pages of records, interviews with the bank, Apple and applicants, and an analysis of underwriting data for about 400,000 New York applicants. The Department concluded that it had found no evidence of unlawful discrimination under fair lending law, and that applications from women and men with similar credit characteristics generally received similar outcomes; the disparities individual couples had seen were traceable to lawful factors such as individual rather than joint credit histories, credit scores, debt and income. But the Department did not treat the matter as harmless. It found that deficiencies in customer service and a lack of transparency had undermined trust in the fairness of the card’s decisions, and it noted the broader difficulty that when a decision cannot be explained, a consumer has no way to tell a lawful outcome from an unlawful one. Goldman and Apple went on to let denied applicants seek help improving their credit, to allow earlier appeals of credit terms, and, later, to add the joint-account option whose absence had shaped the original complaints. The finding, in short, was not that the card was biased. It was that no one had been able to show, at the moment it mattered, that it was not.
What an auditable version would have shown
The question the Department set out to answer, whether outcomes for similar applicants were similar across protected groups, is an aggregate measurement. Its investigation reconstructed that measurement from about four hundred thousand applications over more than a year. An auditable version computes the same measurement continuously and signs it, so the distribution of credit limits by relevant applicant characteristics is a figure the bank, a regulator or a board can inspect at any time, rather than one assembled after the fact. Alongside it, each individual decision would carry a record of the factors that drove it, the kind the customer-service representative could not produce when a cardholder asked why a spouse with a better score received a lower limit. With both, a claim of bias can be met with a checkable answer, and a disparity explained by joint-versus-individual credit history can be told apart from one that is not.
Where the gap was
An automated system set decisions that affected people’s finances, and when those decisions were challenged, neither the customer, the front-line staff nor the public could see the basis for them or verify that the pattern of outcomes was fair. A MetricRecord makes the aggregate behaviour of the model, outcome rates across applicant groups, bound by a chain root hash to the underlying decisions, a signed figure a regulator or board can check, which is precisely the evaluation the Department had to build from scratch. A ConductRecord ties each credit decision to the factors that produced it, so that the reason a particular limit was set is retrievable and explainable at the counter, rather than absent when a cardholder or a representative goes looking for it.
What governance should have looked like
The Department cleared the card, but reached that finding by reconstructing, from hundreds of thousands of applications, evidence the bank did not have ready to hand. In the interval between the viral claim and the finding, the card had no quick way to answer the accusation, because the basis of its decisions was not available in a form a customer, a representative or the public could check. Where an algorithm sets terms that affect people, explainability and verifiable aggregate fairness are what let a claim of bias be answered with a record rather than a reassurance. Best practice would be for an institution running automated decisions about people to be able to demonstrate, on demand and from signed records, both why any single decision was made and how outcomes are distributed across protected groups, so that the fairness of the system can be shown rather than taken on faith or reconstructed later.
Failure Pattern: an algorithm set credit limits that neither customers nor front-line staff could explain, so a fairness question could not be answered from any record and had to be reconstructed by a regulator.
Governance Principle: an automated decision that affects people must be explainable from signed records, and its aggregate outcomes verifiable, so fairness can be demonstrated on demand rather than presumed or denied.
The reference implementation of MetricRecord and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Report on the Apple Card Investigation (New York State Department of Financial Services)
- DFS Issues Findings on the Apple Card and Its Underwriter Goldman Sachs Bank (New York State Department of Financial Services)
- Goldman cleared of bias claims in NYDFS’s Apple Card probe (Banking Dive)
- New York’s Department of Financial Services says Apple Card program didn’t violate fair lending laws (TechCrunch)