What happened
It was reported that in October 2022 Koko, an online peer-support platform, ran a feature that used OpenAI’s GPT-3 to help write emotional-support messages. Volunteers offering support could see a machine-drafted reply and edit it before sending it on. Around 4,000 people were reported to have received messages that were at least partly written by the model.
Reporting indicates the people receiving those messages saw a note that the reply was written in collaboration with Koko Bot, but were given little more than that, and that there was no clear step where they were told a machine was involved and agreed to take part. In January 2023 Koko’s co-founder described the feature publicly. He said the AI-assisted messages had at first been rated more highly than ones written by people alone, and that the effect faded once people learned a machine was involved. Researchers and ethicists said people reaching out for support had in effect been experimented on without informed consent. He defended the work, saying the feature was opt-in, that a person reviewed the messages before they were sent, and that people could skip a message, while arguing that not all uses of AI should need formal ethics-board review. He also accepted that the disclosure had been thin.
What an auditable version would have shown
When a person reaches out for help and a reply comes back, the one thing they are owed is to know who, or what, is writing to them. An auditable version keeps, for each message, whether a model helped write it, what the person was told, and whether they agreed to take part. With that record, the question everyone asked afterwards, were people told a machine was involved and did they consent, has an answer on file. Without it, the disclosure is a single soft label, and the debate becomes one person’s account against another’s.
Where the gap was
The gap was that the disclosure and the consent were not built into the product as something you could check. People got a small label, Koko Bot, and nothing that recorded what they were actually told or whether they agreed to take part. So when the question came later, were these people informed and did they consent, the answer rested on the founder’s account rather than on anything kept at the time.
What governance should have looked like
The more fragile the person reading the reply, the plainer you have to be about who wrote it. Someone reaching out for help should be told, in clear words, that a machine helped write the answer, and given a real chance to say no, before any of this reaches people in distress. A conduct record keeps proof of that for each message: whether the model helped, what the person was told, and whether they agreed. None of this stops anyone using AI to help write support. It just means the people on the other end know, can decline, and the platform can show it treated them fairly.
The reference implementation of ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed, free for any company to install. The repository is public now.
Sources
- ChatGPT used by mental health tech app Koko in AI experiment with users (NBC News)
- A mental health app tested ChatGPT on its users; the founder said backlash was a misunderstanding (Gizmodo)
Related
This entry concerns mental health support. If you or someone you know needs help in Australia, Lifeline is available on 13 11 14.