AI Anatomy Class · Part 11 · Graduation project

92. Designing a mini GPT

Q. If you shrink ChatGPT way down, could you build one with your own hands?

1. Graduation project time! With the parts you've learned, you'll build a real tiny GPT and train it right on your phone.
2. There are four steps: pick the data, build a tokenizer, train the model, and finally let it write.
3. Same blueprint as GPT-3 (chapter 5), but 1 block instead of 96, and token vectors of 32 numbers instead of 12,288.
4. It has 1 attention head and reads 24 tokens at a time. 349 kinds of tokens: 49 characters plus 300 merged pieces (chapter 93).
5. Count the knobs (parameters): GPT-3 has 175 billion, the mini GPT 24,925. That's about 7 million times fewer.
6. But it does the same job: it reads text and gives the odds of the next piece. This is a real answer from a mini GPT we trained.
7. Proverbs in, proverbs out; rhymes in, rhymes out; textbook sentences in, explanations out. A model resembles what it reads.
8. The data is about 3.3 KB, about 1,800 tokens: tiny next to GPT-3's 300 billion, but enough. Shall we pick one?

Chapter summary

• The mini GPT has the same blueprint as GPT-3. Only the numbers of layers, vector sizes and heads are small.
• 24,925 knobs: about 7 million times fewer than GPT-3.
• A model resembles what it reads, so picking the data is step one.