What happened
It was reported that in February 2024, Swedish payments company Klarna announced that its new AI customer service assistant, developed with OpenAI, had handled 2.3 million conversations in its first month. The company claimed the technology was doing the equivalent work of 700 full-time employees, reducing response times and delivering customer satisfaction scores comparable to those achieved by human agents.
Klarna projected that the technology would contribute 40 million USD in additional profit that year, and the announcement quickly became an example of how AI could transform customer service. The comparison with 700 employees attracted particular attention, although the figure represented additional staff Klarna estimated it would otherwise have needed, rather than 700 people directly replaced by the chatbot.
As the company continued expanding its use of AI, however, the limitations of the approach became more apparent. While the assistant could deal with straightforward enquiries about payments and refunds, more complicated situations requiring judgement, reassurance or an understanding of individual circumstances were harder to manage through automation.
By May 2025, Klarna chief executive Sebastian Siemiatkowski was publicly acknowledging that the company’s focus on efficiency had come at a cost to customer service.
“We focused too much on cost,” he said. “The result was lower quality.”
Klarna subsequently announced plans to recruit more human customer service agents and make it easier for customers to speak to a person. The company was not abandoning AI, but recognising that automation alone could not provide the level of service it wanted to deliver.
What an auditable version would have shown
Klarna’s early performance figures told an impressive story about efficiency, but the number of conversations handled and the average time taken to resolve them could not fully explain whether customers were receiving the help they needed.
A chatbot might successfully answer thousands of routine payment questions while struggling with a smaller number of complicated disputes. When those experiences are combined into one overall satisfaction score, problems affecting particular groups of customers can be difficult to identify.
An auditable system would have measured performance separately across different types of enquiries, including payment disputes, fraud concerns and situations requiring more detailed assistance. A MetricRecord could have tracked whether problems were genuinely resolved, while a ConductRecord could have shown when customers needed further support or asked to speak with a person.
The public reporting does not establish exactly which internal measures Klarna used or when the company first identified declining service quality.
Where the gap was
The issue was not that Klarna introduced AI into customer service, because the technology appeared capable of handling a substantial volume of routine enquiries. The difficulty arose when the benefits demonstrated in straightforward interactions were used to support a much broader approach to customer service.
Headlights identifies scope drift, where a system that performs well within a particular set of tasks is given responsibilities that require capabilities it has not demonstrated consistently.
There is a human tendency behind this pattern. When a technology delivers impressive early results, organisations can become increasingly confident in its ability to do more, particularly when those results support financial targets. Measures such as response times, conversation volumes and operating costs are easy to communicate, while the experience of a customer struggling through a complicated financial problem is harder to capture in a single number.
This does not establish that Klarna deliberately overlooked those customers, but it shows why overall efficiency figures can provide an incomplete picture of service quality. A system can appear successful across thousands of interactions while still performing poorly in the situations where customers most need help.
This behavioural interpretation is Headlights’ analysis, not an official finding about Klarna’s internal decisions.
What governance should have looked like
Klarna needed clear boundaries around the types of enquiries its AI assistant could manage reliably, supported by regular reviews of customer outcomes rather than relying primarily on overall efficiency.
Routine questions could be handled automatically, while disputes, fraud concerns and sensitive financial situations should have a straightforward route to human assistance. Customers requesting a person should not have to repeatedly explain their problem before being transferred.
A MetricRecord could identify declining satisfaction or unresolved complaints within particular categories, while a ConductRecord could preserve the reasons for escalation and help reveal where the assistant was struggling. Together, these records would give the company a clearer understanding of which tasks were suitable for automation and which continued to require human judgement.
Klarna’s decision to expand human support demonstrated the importance of that distinction. The lesson is not that AI has no place in customer service, but that handling a large number of conversations is not the same as resolving every customer’s problem.
Failure Pattern: Klarna initially reported substantial efficiency gains from its AI customer service assistant, but its chief executive later acknowledged that prioritising cost reduction had affected service quality. The company subsequently moved to expand human support alongside automation.
Governance Principle: AI customer service should be assessed on the quality of outcomes across different types of enquiries, with clear limits on automation and accessible human assistance when customers need it.
The reference implementation of MetricRecord and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Klarna AI assistant handles two-thirds of customer service chats in its first month (Klarna press release, February 2024)
- Klarna’s AI assistant does the work of 700 full-time agents (OpenAI case study)
- Klarna Reverses AI Customer Service Replacement (Tech.co)
- Klarna reverses AI push, hires customer service agents (eMarketer)
- [Klarna Dials Back its AI Customer Service Strategy