AI Anatomy Class · Part 3 · Turning text into numbers

30. Merging pieces that often go together

Q. GPT-4o knows about 200,000 tokens. Did someone pick every one of them by hand?

1. Nobody writes the token list by hand. We gather lots of text and merge pieces that often appear together. This is BPE.
2. Let's try it on a tiny word list. At first every letter is a piece. Counting how often each word appears, that's 79 pieces.
3. Now count every pair of neighbors. 'e' followed by 's' shows up 9 times, the most of all.
4. Merge the top pair into one piece and add it to the list. Now 'es' is a token, and we're down to 70 pieces.
5. Count again, merge again: est, lo, low. After four merges, the 79 pieces have become 47.
6. Nobody taught it grammar, yet the word 'low' and the ending 'est' became pieces. It only counted.
7. So even a new word like 'lowest' can be cut into known pieces: low, est. And 'slower' becomes s, low, e, r.
8. GPT-4o's tokenizer was made by merging like this, about 200,000 pieces. That's why 'schoolwork' becomes ' school' + 'work'.

Chapter summary

• BPE: in a pile of text, merge the pair of pieces that appears together most often.
• Repeat counting and merging until the list is as big as you want.
• Common chunks become pieces, and even a new word can be cut into known pieces.