AI Anatomy Class · Part 3 · Turning text into numbers

31. Do some languages need more tokens?

Q. People say asking an AI something in Korean costs more than asking in English. Is that true?

1. Let's write the same sentence in four languages and count the tokens. The token count is how much the AI must read and write.
2. With GPT-3's tokenizer, English takes 5 pieces but Korean takes 32. Same meaning, over 6 times as many!
3. See all the dashed pieces? A Korean character is 3 bytes (ch. 28), and this tokenizer cut each one into 2 or 3 byte pieces.
4. GPT-3's tokenizer was built mostly from English text. Of its 50,257 pieces, not one is a whole Korean character.
5. Extra tokens cost you: you pay per token, how much an AI can read at once is counted in tokens, and answers get slower.
6. Newer tokenizers learned far more non-English pieces. In GPT-4o our sentence is 8 tokens in Korean and Japanese, 6 in Chinese.
7. On a long paragraph, GPT-4o needs 1.4 times as many tokens for Korean as for English, 1.3 for Japanese, and 0.9 for Chinese.
8. Do some languages need more tokens? Once far more; now a little more, or the same. It depends on what the tokenizer learned from.

Chapter summary

• The same meaning can take a different number of tokens in each language. Price, reading limit and speed all depend on the token count.
• GPT-3's tokenizer was made from English text, so it chopped Korean characters into bytes.
• GPT-4o learned many more pieces from other languages, so the gap shrank (long text, Korean: about 4.9× → 1.4× English).