AI Anatomy Class · Part 8 · The birth of ChatGPT

76. Polishing with praise

Q. If you can't fully put 'a good answer' into words, how do you teach it?

1. Teaching a puppy to sit, you don't explain in words. You give a treat when it gets it right, and that behavior grows.
2. The model is the same. It writes an answer to a question, and the grader from chapter 75 scores it. That score is the treat.
3. High-scoring answers are made more likely, low-scoring ones less. This is RLHF: reinforcement learning from human feedback.
4. Line up the kinds of answers, and the model creeps toward higher scores, like a puppy toward a treat.
5. But chasing only the score is risky. It can find the grader's blind spot and run toward strange answers full of flattery.
6. So it gets a leash: drift too far from the original model (chapter 73), and it loses points. This leash is the KL penalty.
7. It worked amazingly. People preferred answers from the 1.3 billion InstructGPT, with RLHF, over the 175 billion GPT-3.
8. Copying model answers and praise from people: with these two, chat practice is done. Another blank on the blueprint is filled!

Chapter summary

• RLHF: the model writes answers, and its numbers are adjusted so answers the grader scores highly come more often.
• Chasing only the score leads to strange answers that exploit the grader's blind spots.
• A leash (the KL penalty) keeps it from drifting too far from the original model.