AI Anatomy Class · Part 5 · Mastering attention

48. Divide, then softmax

Q. Why not just put the relevance scores through softmax? Why bother dividing first?

1. In chapter 47, 'school' gave the scores 2, −1, 3, 1. Hey, those are the same numbers as chapter 14's scores 3, 2, 1, −1!
2. Plain softmax gives exactly what chapter 14 gave: to 65.7%, I 24.2%, school 8.9%, walk 1.2%.
3. But attention adds one more step first: it divides the scores by √d, where d is the number of entries in the Q and K vectors.
4. Our example has 2 entries, so we divide by √2, about 1.41. 2 becomes 1.41, −1 becomes −0.71, 3 becomes 2.12, and 1 becomes 0.71.
5. Softmax the divided scores: to 55.7%, I 27.5%, school 13.5%, walk 3.3%. Less lopsided than before.
6. Why divide? Longer vectors make bigger dot products. With 128 entries, scores like 20 or −14 appear, and one word takes almost 100%.
7. That lopsided, nudging a score changes nothing, so learning stalls (ch. 21). Divide by √128 ≈ 11.3 and the shares spread out again.
8. Now we know how much 'school' looks at each word. Next chapter, we mix the Values in these shares.

Chapter summary

• Divide the scores by √d, then softmax. d = the number of entries in Q and K.
• Longer vectors make bigger dot products; left alone, one word takes almost 100%.
• Dividing by √d keeps the shares sensible whatever the length, and learning works well.