AI Anatomy Class · Part 3 · Turning text into numbers

32. The limits of ID numbers

Q. Every token already has a number. Why not just feed those numbers into the model?

1. Every token has a number (ch. 29): ' walk' is 5229 and 'I' is 40. What if we fed these numbers straight into the model?
2. Models multiply and add numbers (ch. 12), so 'walk' would count as 131 times bigger than 'I'. Yet an ID's size means nothing.
3. So instead of a number, we mark a position. Make one slot per word and put a 1 only in your own slot. That's one-hot encoding.
4. No more size differences. But there are two problems. First, it's far too long: for GPT-4o, about 200,000 slots with a single 1.
5. Second, all words are equal strangers. Any two words are √2 apart (ch. 8) with a dot product of 0 (ch. 9).
6. One-hot can't show that a dog and a cat are alike. They're exactly as far apart as a dog and a bus.
7. What we need is a short row of numbers where similar meanings sit close together. That's the embedding, coming next chapter.

Chapter summary

• The size of a token ID means nothing, so we can't use it directly in calculations.
• One-hot: one slot per word, with a 1 only in your own slot. Size differences disappear.
• But it's far too long (about 200,000 slots), and every word is equally far apart (distance √2, dot product 0).