AI Anatomy Class · Part 1 · AI speaks in numbers

15. Matching sizes: normalization

Q. One test is out of 100, another out of 20. Who did better?

1. The math test is out of 100, but the fitness test is out of 20. With different scales, you can't just compare the numbers.
2. So we match their sizes. First, subtract the mean. The mean moves to 0, and you can see what's above or below it.
3. Second, divide by the spread: the standard deviation from statistics class. Now both tests use the same yardstick.
4. Math 85 and fitness 17: both 1.5 standard deviations above the mean. Equally good! Matching sizes like this is called normalization.
5. Inside GPT, numbers can grow or shrink as vectors pass through the layers. So every layer resizes them with normalization.
6. That's the layer normalization we'll see in chapter 57. It keeps the math steady even with dozens of layers stacked.
7. Subtract the mean, divide by the spread. And with that, your Part 1 math toolbox is complete!

Chapter summary

• ① Subtract the mean: the mean moves to 0.
• ② Divide by the standard deviation: the spread becomes 1.
• GPT normalizes at every layer so its numbers don't run wild (chapter 57).