AI Anatomy Class · Part 5 · Mastering attention

49. Mixing to make new meaning

Q. We know how much to look at each word. So what do we bring back with those shares, and how?

1. Chapter 48 gave the shares 'school' looks with: to 55.7%, I 27.5%, school 13.5%, walk 3.3%.
2. Now take out each word's Value, the book's content: a vector holding the information each word hands over. Let's give each a color.
3. Think of mixing paint. Squeeze out each paint by its share and pour it all into one cup. The most paint comes from 'to'.
4. In numbers, multiply each Value by its share and add them all. For example, the share from 'to' is 0.557 × (3, 2). Add up all four.
5. The result is about (2.25, 1.52): a new vector for 'school' with the context mixed in. It holds the most information from 'to'.
6. The shares add up to 1, so the mix always lands somewhere between the Values, pulled toward the words it looked at most.
7. In chapter 45, 'trunk' moved toward the car's trunk with exactly this calculation. Multiplying by shares and adding is a weighted sum.
8. So far we've worked out just one word, 'school'. Next chapter we compute every word at once, with matrices.

Chapter summary

• Multiply each Value by its share and add them all up. That's a weighted sum.
• The shares add up to 1, so the result always lies between the Values.
• It becomes a new vector that holds more of the words it looked at most (ch. 45's 'trunk').