Informing clients January 2023

Koko: emotional support messages written with GPT-3

A volunteer emotional support service sent replies partly written by GPT-3 to about 4,000 people, flagged only by a short note in brackets. In January 2023 it was widely criticised for the lack of informed consent.

Koko is a non-profit service based in San Francisco where volunteers offer each other emotional support, through platforms such as Discord and Tumblr. About 10,000 volunteers helped in a month, and some 1,000 people a day used it.

In October 2022 Koko tested having replies written wholly or partly by the GPT-3 language model. On Discord, helpers could write their own reply or ask for a machine draft, which they could send, edit or discard. The service offered such drafts for about 30,000 posts, the helpers accepted roughly half, and so about 4,000 people received replies at least partly written by the model. Recipients only saw that the reply had been written in collaboration with Koko Bot, not what that meant.

The co-founder described the experiment on Twitter on 6 January 2023. According to him, the co-written replies were rated higher and helpers responded twice as fast, but once people learned that a machine had been involved the benefit disappeared, so the experiment was stopped. The thread drew more than 8 million views and heavy criticism: critics said vulnerable people had been made test subjects without their knowledge or consent.

Research ethics experts said informed consent requires a clear account of risks and benefits; the director of a bioethics centre recommended independent ethics review. The co-founder argued that the experiment was exempt from informed consent because participants had opted in, their identities were hidden and every reply was checked by a person. He told NBC News that the only way for a recipient to opt out was not to read the reply.

What can we learn from it?

The Koko case is not about a data leak but about what the person asking for help knows. Someone writing in a hard moment rightly assumes that a person is answering, and a note in brackets is no substitute for real information. Machine help is not a problem in itself: it can support a professional's work in many ways. Trust depends on whether the client learns in advance, in plain terms, whether a machine plays a part in the work and what happens to their words. That is worth settling in a sentence or two at the start.

Sources

This article is based on the public sources listed. If you find a mistake, write to us: [email protected]