AI Anatomy Class · Part 3 · Turning text into numbers

36. How does it know the order?

Q. '10:11' and '11:10' are made of exactly the same pieces. How does an AI know they're different times?

1. 'Dog bites man' and 'Man bites dog' use the same three words, but who bit whom flips. In English, word order says who did what.
2. Times work the same way. 10:11 and 11:10 are made of exactly the same pieces, only reordered, and they mean different times.
3. Attention (ch. 45) looks at all the pieces at once and mixes them. Without position info, both look like the same bag of pieces.
4. So we add numbers that tell each piece its position. But adding 1, 2, 3 as they are would bury the meaning of piece 2,000.
5. The 2017 Transformer used sine waves: read the height of waves of different speeds at each spot. Values stay between −1 and 1.
6. Read a clock's fast hand and slow hand together and you know the time. Waves work the same way: every spot gets its own pattern.
7. Add this pattern to each piece's vector. Now the same '10' at position 0 and at position 2 gets different vectors, so order is kept.
8. Letters, tokens, IDs, embeddings, position info: text is now numbers a model can compute with. That's Part 3. Next up, Part 4.

Chapter summary

• The same pieces in a different order can mean something else (10:11 ≠ 11:10, 'Dog bites man' ≠ 'Man bites dog').
• Position info: add a pattern that says 'which position' to each piece's vector.
• The pattern is made from sine waves of different speeds (2017). Values stay within −1 to 1 and differ at every position.