AI Anatomy Class · Part 7 · Teaching GPT: pre-training

64. Studying without an answer key

Q. Text on the internet has no answer key. So what did GPT check its answers against?

1. In chapter 4, people had to label each email 'spam' or 'not spam'. But no one can label every text on the internet.
2. The secret: the text itself is the answer key. Let's take one sentence: 'I walk to school.'
3. Show the beginning, hide the next piece, and it's a question. What comes after 'I'? The answer is right there: 'walk'.
4. Add one piece at a time and you get more questions. After 'I walk' comes 'to', and after 'I walk to' comes 'school'.
5. A 5-piece sentence gave us 4 questions. The text, not a person, provides the answers. That's self-supervised learning.
6. The model solves all 4 questions at once. Thanks to chapter 51's mask, each position can only see what comes before it.
7. All of GPT-3's roughly 300 billion tokens are questions like these. About 300 billion questions, without a single answer key.

Chapter summary

• Show only the beginning of a sentence and ask for the next piece: that's a question.
• The answer is the very next piece in the text. No person needs to label anything = self-supervised learning.
• A sentence of N pieces → N − 1 questions. The whole internet becomes a workbook.