AI Anatomy Class · Part 10 · The world after ChatGPT

90. AI in my phone

Q. How can a huge AI run inside my phone, even without the internet?

1. GPT-3 has 175 billion knobs, its parameters (chapter 12). Store each as a 32-bit, or 4-byte, number. How big is that altogether?
2. 175 billion × 4 bytes = 700 billion bytes, about 700 GB. Phones today have about 8 to 12 GB of memory. Not even close!
3. Way one: make a smaller model. Teach a small student model to copy the answers of a big teacher model. This is called distillation.
4. It copies not just the right answer but the teacher's whole set of probability bars, learning that 'the' is fairly likely too.
5. Way two: write numbers less precisely. 8 bits is like a ruler with 256 marks, 4 bits a ruler with 16. This is called quantization.
6. Measure 0.82 with the 8-bit ruler and you get 0.824; with the 4-bit ruler, 0.88. A little off, but the model mostly works the same.
7. Try a 3-billion-parameter model: 12 GB at 32 bits, 3 GB at 8 bits, 1.5 GB at 4 bits. Now it fits on a phone!
8. Running on the phone works without internet, and your text never leaves the phone. But a small model may struggle with hard questions.

Chapter summary

• Model size = number of parameters × bytes per number. GPT-3 at 32 bits is about 700 GB.
• Distillation: a small student model learns by copying a big teacher model's probability bars.
• Quantization: write numbers roughly, in 8 or 4 bits instead of 32. Small model + quantization = AI in your phone.