AI Anatomy Class · Part 3 · Turning text into numbers

35. Exploring the embedding universe

Q. On the word map, who lives next to 'dog', and where does 'trunk' end up?

1. Let's explore the embedding universe we've built. Here are 31 words spread on a 2D map. The positions are an illustrative example.
2. Search for 'dog' and the closest words light up first: rabbit, elephant, cat… all animals.
3. Here 'closest' means distance (ch. 8). Real AI search often compares directions instead, using the dot product (ch. 9).
4. You can also find the odd one out: cat, rabbit, tiger, bus. Only bus sits far away.
5. But where is 'trunk'? It's stuck between the animals, the vehicles and the plants, because it has several meanings.
6. A word gets only one spot, so you need the sentence to know which trunk it is. Attention solves this in chapter 45.
7. Real embeddings have 12,288 dimensions in GPT-3. This map is a 2D shadow of them, so it's a little distorted.
8. In the embedding universe, close means alike and directions show relationships. Words have become numbers with meaning.

Chapter summary

• In embedding space, nearby words have similar meanings. Search = finding the nearest points.
• A word with several meanings, like 'trunk', sits between groups.
• Which meaning it has depends on the sentence. That's attention's job (ch. 45).