AI Anatomy Class · Part 5 · Mastering attention

47. Relevance scores

Q. Can a single number measure how well the question of 'school' fits the label of 'to'?

1. In chapter 46 every word got Q and K cards. Now we score how well the question of 'school' fits each word's label.
2. We use chapter 9's dot product: multiply matching entries and add. The Q of 'school' (1, 2) and the K of 'to' (1, 1): 1×1 + 2×1 = 3.
3. As arrows: the more they point the same way, the bigger the score. The K of 'walk' points elsewhere: 1×1 + 2×(−1) = −1.
4. Work out the rest: 'I' gets 2, and 'school' itself gets 1. These four numbers are the relevance scores 'school' gives.
5. Every other word asks its question too. Four words each checking four words fill a 16-cell score table.
6. Rows are the asking words, columns the answering words. We compare within a row to decide whom to look at. In the 'I' row, 'walk' wins.
7. But some scores are negative, and every row adds up to something different. Next chapter turns them into shares of attention.

Chapter summary

• Relevance score = Q · K (dot product, ch. 9): multiply matching entries and add them all up.
• The more they point the same way, the bigger the score; pointing opposite ways makes it negative.
• When every word scores every word, you get a score table. Row = asking word, column = answering word.