AI Anatomy Class · Part 6 · Assembling the transformer

61. Following it from start to finish

Q. While ChatGPT writes a single word, what exactly happens inside?

  1. Let's feed 'I walk to' into the GPT we've assembled and follow it from start to finish.
  2. First, the tokenizer cuts the text into pieces (tokens). Each piece gets its number in the vocabulary (chapter 29).
  3. The embedding fetches a row of meaningful numbers for each number. In GPT-3, one piece is 12,288 numbers (chapter 33).
  4. Then we add position info that tells which piece comes where (chapter 36). Now 3 vectors head into the block.
  5. Inside the block: normalize, then attention! 'to' looks at 'I', at 'walk', and at itself (chapter 45).
  6. What it mixed in is added to the original vector on the highway (chapter 56). See how the numbers shifted a little?
  7. Normalize again, and feed-forward adds a thought to each word. That result is added to the original vector too (chapter 55).
  8. That's one layer of the block. Repeat it 96 times, and context piles up in every vector (chapter 58).
  9. The 'to' vector from layer 96 scores all 50,257 tokens: school 3.0, the 2.1, work 1.5 (chapter 60).
  10. Softmax turns them into probabilities: school 52%, the 21%, work 12%. Everything adds up to 100% (chapter 14).
  11. Pick one by probability: 'school'! Append it, and the text becomes 'I walk to school' (chapter 1).
  12. Then this text goes back in at the top to make the next piece (chapter 2). Every ChatGPT answer comes from this trip.

Chapter summary