AI Anatomy Class · Part 5 · Mastering attention

45. Words look at each other

Q. Is it an elephant's trunk or a car's trunk? How does an AI tell the difference?

1. The word 'trunk' has several meanings: an elephant's trunk, a car's trunk, and a tree trunk.
2. Chapter 33's embedding gives 'trunk' one fixed vector. On its own, it sits in the middle of the three meanings.
3. Now let's put it in a sentence: 'bags in the car trunk'.
4. 'trunk' looks around at the words before it. The more related a word is, the thicker the line, and the more it looks.
5. Then it mixes in their meanings, by how much it looked. Lots of 'car' and 'bags' moved it toward the car's trunk.
6. What if the sentence changes? 'the elephant's long trunk'. This time it looks hard at 'elephant's' and moves toward the elephant.
7. Each word looks at the other words in its sentence and fixes its own meaning. This is self-attention.
8. But how does it decide how related two words are? Next chapter: Query, Key and Value.

Chapter summary

• The same word can mean different things in different contexts.
• Each word looks at the words before it, and looks more at the closely related ones.
• It mixes in their meanings by how much it looked and fixes its own meaning. That's self-attention.