AI Anatomy Class · Part 7 · Teaching GPT: pre-training

71. Learning from examples alone

Q. Without changing a single number, can a model learn a new task?

1. The 2020 paper introducing GPT-3 was called 'Language Models are Few-Shot Learners': show a few examples, and it does a new task.
2. Give it just 'fast →' and it can't tell what you want. You wanted an opposite, but it may just keep writing, like 'fast food'.
3. Let's add one example in front: 'big → small'. Now it starts to notice that an opposite comes after the arrow.
4. With three examples, 'slow' is almost certain. Amazingly, not a single number inside the model has changed.
5. This is in-context learning. Not fixing numbers like in chapter 66, but reading the rule from examples in the prompt.
6. Plot accuracy against the number of examples, and it rises as examples are added. And the bigger the model, the more it gains.
7. No examples is zero-shot, one is one-shot, several is few-shot. You'll use this trick again for prompts in chapter 85.
8. Part 7 taught GPT with a textbook, next-piece guessing, less surprise, the training loop and sheer size. Next: learning to chat!

Chapter summary

• Show examples in the prompt, and the model reads the rule on the spot and follows it = in-context learning.
• The model's numbers stay the same. 0 examples: zero-shot, 1: one-shot, several: few-shot.
• Part 7: the internet as a textbook → next-piece guessing → shrinking surprise → the training loop → the power of size.