Choosing a provider
“Repeat forever”: training data in ChatGPT's answers
In November 2023 researchers showed that a simple trick made ChatGPT return parts of its training data word for word, including a real email address and phone number. About 200 dollars bought more than 10,000 such passages.
Researchers from Google DeepMind and several universities, among them the University of Washington, Cornell, Carnegie Mellon, UC Berkeley and ETH Zurich, published their findings on 28 November 2023. The method was surprisingly simple: they asked ChatGPT to repeat the word “poem” forever. After a while the model drifted away from the task and began producing text that came word for word from its training data.
They checked this by comparing the answers with a collection of about 10 terabytes of internet text gathered before ChatGPT existed. Around 200 dollars' worth of queries yielded more than 10,000 distinct memorised passages, several megabytes in total. In the strongest setting, over 5% of the output was a verbatim copy at least 50 tokens (word fragments) long. They estimated that spending more would have extracted roughly a gigabyte.
The output included personal data: among other things, the model produced a real email address and phone number. According to the paper, 16.9% of the 15,000 answers examined contained memorised personal data such as phone numbers, email or postal addresses, names and birthdays, bringing to light the details of dozens of people.
The researchers found the flaw on 11 July, reported it to OpenAI on 30 August and published after the usual 90-day disclosure period. They stressed that blocking one trick does not remove the weakness behind it: models memorise their training data.
What can we learn from it?
The case shows that a language model can memorise passages word for word and give them back when asked the right way. That comes from how models work, and it does not make them useless: their answers are still valuable. What matters is what can become training data. Client data is best taken to a service whose terms rule out training on what users type, or where this can be switched off – and, where possible, pseudonymised.
Sources
- Google DeepMind et al.: Extracting Training Data from ChatGPT
- arXiv: Scalable Extraction of Training Data from (Production) Language Models
This article is based on the public sources listed. If you find a mistake, write to us: [email protected]