The 13 behaviours
When AI goes wrong, the conversation usually stops at the technology. What did the system do wrong? But in almost every incident I document, people made choices too: what they trusted, what they checked, what they ignored and what they did when it went wrong. I read each incident against thirteen behaviours, each one has a test the public record has to meet, if the record does not show it, I do not assign it
They trusted something they hadn't established.
An organisation claimed its AI could do something, or relied on it as though its accuracy or safety had been established. But the evidence wasn't there to support that confidence.
It's the behaviour I find most often.
Example: Zillow relied on its home-pricing algorithm to buy thousands of houses, then said the unpredictability in forecasting prices "far exceeds what we anticipated" and ended its home-buying business. HD-INC-017
The rules said one thing, what happened was another
A policy, procedure or law existed but people did not follow it
Having a rule and following it are two different things.
Example: Kmart scanned shoppers' faces in 28 stores. The privacy regulator found it collected their biometric information without consent, in breach of Australia's privacy principles. HD-INC-021
They trusted the computer when they had reason to question it.
The system produced an answer and someone accepted it as correct. The information needed to question that answer was available, but it wasn't used.
The problem wasn't simply trusting technology. It was accepting its answer when there was reason to check.
Example: Detroit police ran a facial recognition search and put the person it returned into a photo lineup. That lineup was shown to someone who had not witnessed the theft. A warrant followed. The city's later settlement bars arrests based on a facial recognition match alone. HD-INC-054
The human question: Why was the computer's match allowed to carry so much weight when the person looking at the photographs hadn't even seen the crime?
A decision that needed human judgement was handed to AI.
Some decisions require a person to consider the circumstances and make a judgement.
Here, the AI made the decision without a human's involvement, or the human involvement did not amount to someone genuinely exercising that judgement.
Example: It was reported that Cigna's doctors signed off claim denials in bulk, about 1.2 seconds each, without opening patient files. Cigna said this mischaracterised the system. HD-INC-064
When something that breaks the rules happens often enough, people start treating it as normal.
A rule is broken once, then again, then again, and over time the departure becomes part of how things are done, with nobody stopping to reconsider whether it should be happening at all.
The term comes from research into the Challenger space shuttle disaster.
Example: Australia's Robodebt scheme kept raising automated welfare debts for years, and the Royal Commission found that it was launched and continued despite repeated legal advice that it was unlawful. HD-INC-009
Information was kept from people entitled to see it.
An organisation withheld how a system operated, how it worked or what it had done. Sometimes the means of checking afterwards were removed.
I only use this behaviour when someone entitled to the information was refused it, or when the withholding itself was found or admitted.
Example: When campaigners asked for the country classifications behind the UK Home Office's visa grading tool, they received a list with the country names redacted. HD-INC-153
The problem was known. Action didn't follow for some time.
Someone knew about the problem, or it had been reported.
But the organisation did not act for a significant period. This behaviour is about the time between knowing something was wrong and doing something about it.
Example: After an investigation showed New York City's business chatbot giving wrong answers about the law, the city added a "beta" label instead of taking it offline. It was switched off almost two years later. HD-INC-006
It wasn't clear who had the authority to decide.
Nobody had clearly established who was entitled to give an instruction, make a decision or approve an action.
Despite that uncertainty, the system or the people involved went ahead.
Example: A finance worker at Arup took instructions on a video call from what looked like senior colleagues, made transfers of about 25 million US dollars, and only checked with head office afterwards. Everyone on the call was a deepfake. HD-INC-039
They knew there was a risk, but carried on anyway.
Sometimes people inside an organisation know something is wrong, or have been warned that it could be. Yet the organisation continues down the same path.
The important question isn't why nobody spoke up. It's why the organisation carried on when the warning signs were already there.
Example: A public inquiry found that senior Post Office employees knew, or should have known, that its Horizon system could make errors. Yet more than 700 branch operators were prosecuted based on what Horizon reported. HD-INC-116
When something went wrong, the organisation denied it or played it down.
After a failure, an organisation denied what happened, minimised it or refused to reconsider.
I only assign this behaviour when that response is documented. I don't try to guess what anyone was thinking.
Example: Air Canada argued its chatbot was "a separate legal entity", which the tribunal called "a remarkable submission". HD-INC-001
Speed, cost, targets or commercial advantage shaped the decision.
Sometimes the record shows that the pressure to move faster, spend less, meet a target or gain a commercial advantage influenced what happened.
I only assign this behaviour when that influence is documented.
Example: The Federal Court's judgment records that one reason an insurer approved capping its loyalty discounts was the profit it would otherwise lose. HD-INC-078
Everyone assumes someone else is responsible.
When several people or organisations are involved, it can become unclear who is supposed to make the decision or check the work. Each may assume someone else has taken care of it. In the end, nobody clearly owns the responsibility.
Example: Sports Illustrated published articles under author names that belonged to no real person. When questions were raised, its publisher said the content had come from a third-party vendor and ended the partnership. HD-INC-038
They considered the evidence that supported their conclusion, but overlooked what challenged it.
Sometimes we become so convinced we're right that we fail to properly consider evidence pointing the other way. In these incidents, I look for proof that contradictory evidence was available but wasn't weighed before reaching a conclusion.
I haven't yet found enough documented evidence to assign this as the main behaviour in an incident.
Example: No incident currently has confirmation bias as its main behaviour.
A note about these behaviours
I'm not trying to work out what people were thinking. I wasn't in the room, and I don't know what motivated their decisions.
What I can do is look at what happened, what people knew at the time, and what they did next.
Every behaviour has an evidence test. If the published record doesn't support it, I mark it not established. These are my research labels, not psychological diagnoses or findings by a court or regulator.
I research and maintain this library independently. It isn't peer reviewed, and the numbers don't tell us how common these behaviours are across all AI systems.
But I think they're worth examining.
We spend a lot of time asking why AI systems fail. We spend far less time asking why people trusted them, ignored warnings, or failed to step in.
And sometimes, that's where the most important part of the story is.
“We cannot change the human condition, but we can change the conditions under which humans work.”