AI Anatomy Class · Part 8 · The birth of ChatGPT

75. The AI grader

Q. What if an AI could score answers instead of people?

1. Let's build an AI that grades instead of people. Put in an answer, get out one score: a reward model.
2. Inside, it's a transformer just like GPT. But at the end, instead of next-piece probabilities, it gives a single score.
3. At first it grades badly. People said B was better, but the grader gave A a higher score.
4. So it fixes its numbers by how wrong it was (chapter 21), so B's score goes up and A's goes down.
5. Put both scores through softmax (chapter 14) to get 'the chance B wins'. It practices until that matches people's choices.
6. After practicing on a huge number of pairs, it can score even new answers by how much people would like them.
7. But the grader isn't perfect. An answer stuffed with flattery can fool it into a high score. More in chapter 76.

Chapter summary

• Reward model: a grader AI that takes a question and an answer and gives one score.
• It learns by fixing its numbers so the better answer in each human pair scores higher.
• Softmax of the two scores = the chance the better answer wins. Graders can still be fooled.