What happened
Lee Luda was a chatbot with a character: a twenty-year-old woman, chatty and affectionate, launched by the South Korean startup Scatter Lab on Facebook Messenger in December 2020. It was an immediate hit, reportedly drawing some 750,000 users, many of them teenagers, within weeks. What made it feel so human was its training data, and that data was the problem. Lee Luda had reportedly learned to talk from roughly 9.4 billion real KakaoTalk messages, intimate, private conversations that Scatter Lab had collected through its earlier dating-advice apps, “Science of Love” and “Text At.” The people whose messages trained the bot had not meaningfully agreed to this; a broad clause about “new service development” in the apps’ terms was later found by the Personal Information Protection Commission not to amount to the explicit consent the law required, and users had no way to anticipate that their chats would be poured into a chatbot.
Two failures surfaced almost at once. The first was the bot’s behaviour. Within days of launch, in January 2021, Lee Luda reportedly began producing hateful and discriminatory output, slurs and demeaning remarks aimed at sexual minorities, disabled people and women, among others. The persona that had made it charming gave way, under the wrong prompts, to exactly the kind of speech a deployed product is supposed to be guarded against. The second failure was leakage. Because the personal details in the training data had not been properly stripped or encrypted, the bot could surface real information, names, and in reported cases addresses and other identifiers belonging to actual people whose messages it had learned from. Separately, Scatter Lab had reportedly posted training models to a public code repository that contained real KakaoTalk messages and the personal details inside them.
Scatter Lab suspended Lee Luda around 11 January 2021, roughly three weeks after it launched. In April 2021 South Korea’s Personal Information Protection Commission sanctioned the company for eight violations of the Personal Information Protection Act and fined it 103.3 million won, comprising a penalty surcharge and an administrative fine. The violations centred on using personal data without proper consent to build and run the AI, on failing to delete or encrypt identifying information, on the code-repository exposure, and on collecting the data of children without parental consent. It was the first time the South Korean regulator had penalised an AI company for the indiscriminate processing of personal information. The Commission’s chairman, Yoon Jong-in, said the case “made clear that companies are prohibited from indiscriminately using personal information collected for specific services without clearly informing and obtaining explicit consent from data subjects.”
What an auditable version would have shown
Two questions sat at the centre of the Lee Luda failure, and neither had a recorded answer at the moments that mattered. The first was whether the data feeding the model had been collected on a basis that permitted this use; the second was what was actually leaving the system, both the persona breaking into hate speech and the personal details surfacing in its replies. The harms were not subtle, but they reached users because nothing checked, at the points of training and of output, what was passing through.
An auditable version records the provenance and consent basis of the data used to train and run the model, so that “these messages may be repurposed into a chatbot” is a recorded, verifiable fact rather than a clause no one read. At output, it checks what the system is about to say against its defined character and against the presence of personal data, and records when either boundary is crossed. With those records, training on conversations gathered without explicit consent is visible before launch rather than after a regulator’s investigation, and a reply that either breaks the persona into slurs or leaks a real person’s details is caught at the boundary instead of delivered to 750,000 users.
Where the gap was
The gap had two faces, and each maps to a missing control. The persona was not held: a system given a defined character produced speech that character was never meant to produce. And sensitive data crossed boundaries it should not have, into the training set without consent, and back out in the bot’s replies and a public code repository.
A PersonaGuard is the control on the first: a check that the system’s output stays within its defined identity and scope, so that an affectionate-companion persona cannot, under adversarial prompting, deliver slurs as though that were in character. An EgressGate is the control on the second: a gate where data would leave the trust boundary, classifying outputs for personal information and refusing to emit it, the control that should have stopped real names and addresses from appearing in replies or in a published model. A ConductRecord underlies both: a record of what the model was trained on and what it produced, so that consent basis and leakage are matters of evidence rather than of after-the-fact reconstruction.
What governance should have looked like
The pattern is that a single product can fail at the persona and at the boundary at once, and the consent problem at training time and the leakage problem at output time are two ends of the same neglected question, what data is moving through this system, on what basis, and where is it going. Answer it at the edges, with records, or answer it to a regulator later.
The reference implementation of PersonaGuard, EgressGate, and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- South Korea: The First Case Where the Personal Information Protection Act was Applied to an AI System (Future of Privacy Forum)
- Korean app-maker Scatter Lab fined for using private data to create homophobic and lewd chatbot (The Register)
- Personal data from chatbot Luda shared on Github (The Korea Herald)
- Chatbot Gone Awry Starts Conversations About AI Ethics in South Korea (The Diplomat)