AI Anatomy Class · Part 5 · Mastering attention

50. Computing it all at once

Q. With 1,000 words, do we really have to repeat chapters 47–49 a thousand times, one word at a time?

1. So far we've worked out just one word, 'school'. With 1,000 words, would we have to do this 1,000 separate times?
2. Stack the word vectors row by row into a matrix (ch. 10). Multiply it by chapter 46's three matrices: every word's Q, K and V at once.
3. Next, Q × Kᵀ. Kᵀ is K turned on its side. Each cell of the result is a Q row dotted with a K row: chapter 47's score table.
4. Divide the whole score table by √2 and softmax each row (ch. 48). Now each row holds the shares that word looks with.
5. Finally, the share table × V. Each row of the result is chapter 49's paint mixing: every word's new vector at once.
6. Written as one formula: softmax(QKᵀ/√d)V. That's exactly the attention formula in the 2017 paper (ch. 44).
7. A GPU computes all the cells of a matrix product at the same time. That's how transformers handle long sentences fast (ch. 44).

Chapter summary

• Stack the word vectors into a matrix and make every Q, K and V at once.
• Q × Kᵀ = score table; ÷ √d and softmax each row = share table; share table × V = the new vectors.
• softmax(QKᵀ/√d)V: since it's matrix products, a GPU computes every word at the same time.