AI Anatomy Class · Part 3 · Turning text into numbers

33. Embeddings: coordinates of meaning

Q. How can numbers show that 'dog' and 'cat' are alike?

1. Instead of one-hot, each word gets a short row of numbers: dog is [0.8, 0.7, −0.2]. Turning words into rows like this is embedding.
2. Three numbers make one point in 3D space. The words float in space like stars in constellations.
3. Words with similar meanings gather close together: animals with animals, fruit with fruit. It's chapter 8's word map, in 3D.
4. The dot product measures how alike they are (ch. 9). dog·cat is 1.11, dog·bus is −0.58. The more they point the same way, the bigger.
5. Inside the model is an embedding table, one row per token. Use the token's number to find its row and take it out.
6. At first the table is filled with random numbers. As the model practices guessing the next piece, similar words find their places.
7. How big is the real one? GPT-3 has 12,288 numbers per token and about 50,000 rows: over 600 million numbers in this table alone.
8. Embedding turns tokens into coordinates of meaning: the blueprint's second part is in. Next, how do coordinates get their meaning?

Chapter summary

• Embedding: giving every token a short row of numbers (a vector). The numbers are its coordinates.
• Words with similar meanings sit close together and point the same way (distance ch. 8, dot product ch. 9).
• The numbers in the embedding table are set by training. GPT-3's table holds over 600 million numbers.