AI Anatomy Class · Part 6 · Assembling the transformer

60. From vector to word

Q. After 96 layers, out comes a vector of 12,288 numbers. How does that turn into the word 'school'?

1. After all 96 layers, each word comes out as one vector. The one that decides the next word is the vector of the last word, 'to'.
2. It's compared with every token in the vocabulary to give each a score. Similar direction, higher score (chapter 9, dot product).
3. The vocabulary has about 50,000 tokens, 50,257 to be exact. So there are 50,257 scores too. These scores are called logits.
4. school 3.0, the 2.1, work 1.5, pizza −1.0. The other 50,000 or so mostly score around −9.
5. Now chapter 14's softmax! Raise e to each score, then divide by the sum over all 50,000. That sum is about 38.72.
6. school 52%, the 21%, work 12%. The other 50,000-odd tokens each add a tiny bit, and together they take 14%.
7. Now pick one by probability and append it to the text (chapters 1 and 2). Output + softmax is filled in on the blueprint!

Chapter summary

• The next word is decided by the last word's vector from the final layer.
• It's compared with each of about 50,000 tokens (50,257) to get scores (logits): one 12,288 × 50,257 matrix multiplication (ch. 11).
• Softmax (ch. 14) turns them into probabilities, and one is picked and appended. school 52%, the 21%, work 12%.