AI Anatomy Class · Part 11 · Graduation project

93. Building my own tokenizer

Q. How do you decide the tokens that fit your own data best?

1. The model can't read text directly, only token numbers (chapter 29). The simplest way: make each character one token.
2. Cut the proverbs into characters and you get 3,380 tokens of 49 kinds. Spaces and line breaks are tokens too.
3. BPE counts pairs that sit side by side inside words (chapter 30). In the proverbs the most common pair is 't' and 'h', 72 times.
4. Merge the top pair into one token. The kinds go up by one to 50, and the total drops by 72 to 3,308 tokens.
5. Repeat it. After 30 merges there are 2,686 tokens. The same text now fits in fewer tokens.
6. Is more merging always better? Fewer tokens, but more kinds, so more embeddings to learn. GPT-3 has about 50,000 kinds.
7. Chapter 94's mini GPT uses the tokenizer you build here. No merges means one character per token; our ready-made models used 300.

Chapter summary

• At first, 1 character = 1 token. The proverbs: 3,380 tokens of 49 kinds.
• BPE: merging the most common pair adds one kind and cuts the number of tokens.
• Chapter 94's model uses my tokenizer. Letters say little alone, so English likes many merges.