AI Anatomy Class · Part 3 · Turning text into numbers

29. Tokens: the AI's unit of text

Q. Does ChatGPT read one letter at a time, or one word at a time?

1. Before ChatGPT reads text, it cuts it into small pieces. The pieces are tokens, and the part that cuts is the tokenizer.
2. What if we cut letter by letter? We'd get far too many pieces, and one letter like 'l' means almost nothing.
3. Word by word, then? There are endless words, and new slang like 'rizzler' isn't on any list, so it can't be read.
4. So we cut somewhere in between. GPT-4o's real tokenizer cuts this sentence into 5 pieces. The space sticks to the next word.
5. Every piece has a number: ' school' with its space is 3474. What the model really gets is this row of numbers, not text.
6. The list of pieces is called the vocabulary. GPT-3 knows about 50,000 pieces, GPT-4o about 200,000.
7. A rare word is split into pieces it knows: 'unbelievable' is 3 pieces, 'rizzler' 3 as well. So no word is unreadable.
8. The tokenizer cuts text and numbers the pieces: the blueprint's first part is in. But how is the list of pieces made?

Chapter summary

• Token: the piece of text an AI reads. Often bigger than a letter and no bigger than a word.
• Common chunks are one piece; rare words become several small pieces.
• Every piece has a number, and the model receives a row of numbers. The list of pieces = the vocabulary.