AI Anatomy Class

Learn how ChatGPT was built in 96 short animated chapters with hands-on experiments, written for high school students. Tokens, embeddings, attention, training and RLHF.

  1. 1. The next-word guessing game When you ask ChatGPT something, does it think up the whole answer at once?
  2. 2. An AI that writes one word at a time Why does ChatGPT's answer show up bit by bit, as if someone were typing?
  3. 3. 70 years of AI on a timeline Did AI suddenly appear out of nowhere along with ChatGPT?
  4. 4. Rules vs data How could you build a program that filters out spam texts?
  5. 5. The blueprint we will build If you opened up the machine called ChatGPT, what would you find inside?
  6. 6. Everything is a number How does a computer remember photos, songs and letters?
  7. 7. Vectors: rows of numbers Can you put into numbers how similar your taste in food is to a friend's?
  8. 8. Close means similar Can numbers help you find a friend with a taste like yours?
  9. 9. Dot product: how much the same direction? Can one number tell how much two arrows point the same way?
  10. 10. Matrices: bundles of vectors Is there a way to handle many vectors at once?
  11. 11. Matrix multiplication = transforming space What happens when you multiply a vector by a matrix?
  12. 12. Functions are input→output machines People say an AI model is really just a "function". What does that mean?
  13. 13. Probability distributions How would you say "it'll probably rain" with numbers?
  14. 14. Scores into probabilities: softmax A model's scores can be negative, and they add up to anything. How do we turn them into probabilities?
  15. 15. Matching sizes: normalization One test is out of 100, another out of 20. Who did better?
  16. 16. A single neuron A new taco truck just opened. How would an AI decide whether it's a hit?
  17. 17. Bend to survive: activation functions Tacos are best when they're spicy, but not too spicy. Can a neuron figure that out?
  18. 18. Stacking layers A problem one neuron can't solve: can a group of neurons solve it?
  19. 19. The forward pass When you put numbers into a neural network, what happens inside?
  20. 20. How wrong was it? Loss We know when an AI is wrong, but how do we measure how wrong it is?
  21. 21. Gradient descent To shrink the loss, which way should you turn a knob, and by how much?
  22. 22. Choosing the step size: learning rate Walking down a mountain, how big should each step be?
  23. 23. Backpropagation: sharing the blame With billions of knobs, when the answer is wrong, which knobs do you fix, and by how much?
  24. 24. Splitting the workbook A workbook with millions of problems: fix after every problem, or solve them all and fix once?
  25. 25. The student who memorized past exams An AI that aces every practice problem but fails the real test: why does that happen?
  26. 26. Hands-on: handwritten digits Can a neural network really read a digit you write with your finger?
  27. 27. Part 2 boss battle Put together everything so far: can we train a neural network from scratch?
  28. 28. Letters inside a computer Every letter has a number inside the computer. But who picked those numbers, and does anyone have to memorize them?
  29. 29. Tokens: the AI's unit of text Does ChatGPT read one letter at a time, or one word at a time?
  30. 30. Merging pieces that often go together GPT-4o knows about 200,000 tokens. Did someone pick every one of them by hand?
  31. 31. Do some languages need more tokens? People say asking an AI something in Korean costs more than asking in English. Is that true?
  32. 32. The limits of ID numbers Every token already has a number. Why not just feed those numbers into the model?
  33. 33. Embeddings: coordinates of meaning How can numbers show that 'dog' and 'cat' are alike?
  34. 34. Neighbors reveal meaning Nobody ever taught it what words mean. So how does an AI know that 'coffee' and 'tea' are alike?
  35. 35. Exploring the embedding universe On the word map, who lives next to 'dog', and where does 'trunk' end up?
  36. 36. How does it know the order? '10:11' and '11:10' are made of exactly the same pieces. How does an AI know they're different times?
  37. 37. Language model = probability of a sentence How does an AI know that one sentence sounds 'natural' and another doesn't?
  38. 38. Looking at only the last few words How did AI find 'the chance of the next word' in the old days?
  39. 39. The neural language model How can an AI guess the next word in a sentence it has never seen before?
  40. 40. RNN: passing notes in a relay Sentences come in every length. How does an AI remember what came before while it reads?
  41. 41. The limits of RNNs If the note relay is so good, why doesn't ChatGPT use an RNN?
  42. 42. How a translator is built Two languages with different word orders and lengths. How does an AI translate between them?
  43. 43. The birth of attention What if a translator could glance back at the original while it writes? How much better would AI translation get?
  44. 44. Attention Is All You Need The paper that became ChatGPT's backbone claimed 'attention is all you need'. Why?
  45. 45. Words look at each other Is it an elephant's trunk or a car's trunk? How does an AI tell the difference?
  46. 46. Query · Key · Value How does a word decide which other words in the sentence to look at?
  47. 47. Relevance scores Can a single number measure how well the question of 'school' fits the label of 'to'?
  48. 48. Divide, then softmax Why not just put the relevance scores through softmax? Why bother dividing first?
  49. 49. Mixing to make new meaning We know how much to look at each word. So what do we bring back with those shares, and how?
  50. 50. Computing it all at once With 1,000 words, do we really have to repeat chapters 47–49 a thousand times, one word at a time?
  51. 51. No peeking at the future If you're supposed to guess the next word but can see it in advance, are you really learning anything?
  52. 52. Looking with many eyes 'school' wants to know who's coming and how. Is one look enough for all of that?
  53. 53. Peeking inside a real GPT Are the heads of a real GPT actually 'looking' at anything?
  54. 54. The transformer block blueprint The 'transformer block' sits in the middle of the chapter 5 blueprint. What's inside if you open it?
  55. 55. The thinking layer Attention let the words share information. What does each word do next?
  56. 56. The information highway After 96 layers, wouldn't the information in the original sentence be wiped out?
  57. 57. Keeping things steady If results keep getting added, won't the numbers be huge by layer 96?
  58. 58. Stacking blocks One block already works. So why stack 96 of them?
  59. 59. Encoder vs decoder The 'GPT' in ChatGPT and Google's 'BERT' are both transformers. So what's the difference?
  60. 60. From vector to word After 96 layers, out comes a vector of 12,288 numbers. How does that turn into the word 'school'?
  61. 61. Following it from start to finish While ChatGPT writes a single word, what exactly happens inside?
  62. 62. Part 6 boss battle: the finished blueprint We've collected every part. Can you assemble them in the right order and complete GPT?
  63. 63. The internet is the textbook What books did ChatGPT actually study?
  64. 64. Studying without an answer key Text on the internet has no answer key. So what did GPT check its answers against?
  65. 65. Measuring surprise A model that bet 52% on the right answer: how wrong was it? Can we measure wrongness with a single number?
  66. 66. One lap of the training loop In what order does GPT study hundreds of billions of questions?
  67. 67. A smarter way downhill Why does a ball that only follows the slope get lost?
  68. 68. Why GPUs? Why do AI companies buy piles of 'graphics cards' instead of regular computers?
  69. 69. GPT-1 → 2 → 3 From GPT-1 to GPT-3, how much did it grow in just two years?
  70. 70. Bigger is smarter Make a model 10 times bigger: how much better does it get? Can we know in advance?
  71. 71. Learning from examples alone Without changing a single number, can a model learn a new task?
  72. 72. Smart but doesn't listen Ask a question to a model that has read a huge amount of internet text. Will it simply answer?
  73. 73. Copying model answers How do you teach a model that doesn't listen how to answer?
  74. 74. People rank the answers Is there a much easier way than writing model answers by hand?
  75. 75. The AI grader What if an AI could score answers instead of people?
  76. 76. Polishing with praise If you can't fully put 'a good answer' into words, how do you teach it?
  77. 77. A chat is really one long document How does ChatGPT tell what I said from what it said?
  78. 78. November 30, 2022 How did ChatGPT gather a million users in five days?
  79. 79. Making AI safe Which requests should an AI refuse? And what if it refuses too much?
  80. 80. Ways to choose Why not just pick the most likely word every time?
  81. 81. The creativity dial Can you set an AI's 'creativity' with a single number?
  82. 82. Narrowing the candidates If the wheel has a 'pizza' slice, won't 'pizza' really come up someday?
  83. 83. The AI's memory limit In a long chat, ChatGPT sometimes forgets what you said at the start. Why?
  84. 84. Convincing lies If an AI says something confidently, is it true?
  85. 85. Prompts make sense once you know the structure Why does the answer change with how you ask, even when you ask for the same thing?
  86. 86. AI with eyes How can an AI that only read text look at a photo and describe it?
  87. 87. Think, then answer When you give an AI a hard problem, it shows 'Thinking…'. What is it doing all that time?
  88. 88. Look it up, then answer How do you ask an AI about something it never learned, like the date of your school festival?
  89. 89. AI with hands and feet How can an AI that only writes text punch a calculator or even search the web?
  90. 90. AI in my phone How can a huge AI run inside my phone, even without the internet?
  91. 91. AI and us Who owns a picture an AI drew? And what should I get ready for in the age of AI?
  92. 92. Designing a mini GPT If you shrink ChatGPT way down, could you build one with your own hands?
  93. 93. Building my own tokenizer How do you decide the tokens that fit your own data best?
  94. 94. Training in the browser How does a model really learn? Watch it happen inside your own phone.
  95. 95. Let's generate What will the model I trained write?
  96. 96. Graduation day Shall we follow the ChatGPT we assembled over 96 chapters from start to finish in one go?