AI Anatomy Class · Part 5 · Mastering attention

52. Looking with many eyes

Q. 'school' wants to know who's coming and how. Is one look enough for all of that?

1. One sentence holds several kinds of relationships: who's doing it, what they're doing, and where they're headed.
2. But one round of attention sets just one set of shares. So we make several sets of Q, K and V. Each set is called a head.
3. Say head 1 looks for 'who', head 2 for 'what', and head 3 for 'where'. The same 'school' looks in different places.
4. The heads don't wait for each other: they compute at the same time, separately. Each one hands out its own new vector for 'school'.
5. The results are joined side by side and mixed into one with one more matrix. What many eyes saw ends up in one vector.
6. GPT-3 has 96 heads in every layer, each 128 entries long. 96 × 128 = 12,288, exactly the size of a word vector.
7. Nobody decides what each head looks at. The jobs split up on their own during training. In chapter 53 we look at real heads.

Chapter summary

• One set of Q, K, V = one head. Each head looks at a different kind of relationship.
• The heads compute at the same time; their results are joined and mixed once more with a matrix.
• GPT-3: 96 heads per layer × 128 entries = 12,288. The jobs split up on their own during training.