AI Anatomy Class · Part 6 · Assembling the transformer

57. Keeping things steady

Q. If results keep getting added, won't the numbers be huge by layer 96?

1. In chapter 56, results keep getting added. As they pile up layer after layer, the vector's numbers keep growing.
2. Same word 'to', yet a different size at every layer. The parts would need a new ruler each time, and the math gets shaky.
3. So right before each part, we use chapter 15's normalization: the mean and spread of the numbers inside one word vector.
4. Subtract the mean 10 and divide by the spread 5. The vector that grew 4 times bigger ends up exactly the same.
5. This is done for each word separately. Each word uses only its own 12,288 numbers to find the mean and spread.
6. Last, each number is multiplied by a scale and gets a shift. These are learned, so the model picks the right size itself.
7. GPT normalizes right before attention and right before feed-forward, and leaves the original values on the highway alone.

Chapter summary

• As results pile up layer after layer, the vector's numbers grow and become uneven.
• Layer normalization: each word subtracts the mean of its own numbers and divides by their spread (ch. 15), then gets scaled and shifted.
• GPT normalizes right before attention and right before feed-forward.