AI Anatomy Class · Part 3 · Turning text into numbers

34. Neighbors reveal meaning

Q. Nobody ever taught it what words mean. So how does an AI know that 'coffee' and 'tea' are alike?

1. 'I drank a hot ___ at a café.' What fits in the blank? Even in a sentence you've never seen, the words around it give it away.
2. Words that fit the same spot, like coffee, tea and cocoa, mean similar things. This idea is called the distributional hypothesis.
3. Word2Vec (2013) turned this idea into training: look at one word and guess the words around it, millions and millions of times.
4. After practice, words with similar neighbors end up with nearby vectors, even though nobody told it any meanings.
5. Amazingly, directions carry meaning too. The arrow from man to woman is almost the same as the arrow from king to queen.
6. So king − man + woman lands right next to queen: (7, 2.5) − (2, 2) + (2, 5) = (7, 5.5).
7. Seoul − Korea + Japan ≈ Tokyo too. Real Word2Vec has hundreds of dimensions and isn't always this neat. This map is illustrative.
8. Neighbors reveal meaning. GPT's embeddings learn meaning the same way, by guessing the next piece.

Chapter summary

• Distributional hypothesis: words that appear in the same spots (with the same neighbors) have similar meanings.
• Word2Vec (2013): builds word vectors by practicing guessing the neighboring words.
• Directions carry meaning too: king − man + woman ≈ queen.