AI Anatomy Class · Part 4 · Before the transformer

37. Language model = probability of a sentence

Q. How does an AI know that one sentence sounds 'natural' and another doesn't?

  1. Look at two sentences: 'I walk to school' and 'I walk to pizza'. Which one sounds more natural?
  2. A language model is a machine that gives every sentence a probability: how likely it is to show up. ChatGPT is a language model too.
  3. But we can't count whole sentences. There are far too many sentences nobody has ever seen. So we break them into single words.
  4. The chance the first word is 'I', then that 'walk' comes next… Each time, look at the words so far and find the next word's chance.
  5. Multiply these chances in order and you get the sentence's probability: 20% × 25% × 10% × 52% = 0.26%.
  6. The 'pizza' sentence has only 1% at the end, so it gets 0.005%. Same start, but one last word makes a 52-fold difference.
  7. So a model that's good at guessing the next word is also good at scoring sentences. That's chapter 1's next-word game.
  8. So where do these probabilities come from? The easiest way is to count. We'll count them ourselves in chapter 38.

Chapter summary