AI Anatomy Class · Part 8 · The birth of ChatGPT

74. People rank the answers

Q. Is there a much easier way than writing model answers by hand?

1. Writing model answers is hard, but picking the better of existing answers is easy, just as grading is easier than writing.
2. So the model writes several answers to one question. Picking by probability makes each one different (chapter 2).
3. A person reads them and ranks them: B first, C second, and A, with its bad advice, third.
4. The ranking becomes pairs: B beats C, B beats A, C beats A. That's what real training data looks like.
5. With 4 answers, you get 6 pairs. One ranking gives you several pieces of comparison data.
6. There are three ranking rules: Is it helpful? Is it true? Is it harmless? What gets collected is called human preference data.
7. But people can't grade every answer forever. So this data is used to build an AI that grades. That's the next chapter.

Chapter summary

• Choosing among several answers is easier and faster than writing one yourself.
• A ranking turns into pairs (better answer > worse answer): 3 answers make 3 pairs, 4 make 6.
• The rules: helpful? true? harmless? This is human preference data.