AI Anatomy Class · Part 7 · Teaching GPT: pre-training

68. Why GPUs?

Q. Why do AI companies buy piles of 'graphics cards' instead of regular computers?

1. GPT's math is mostly matrix multiplication (chapter 11). A single GPT-3 layer has several 12,288 × 12,288 matrices.
2. Let's multiply two small 4 × 4 matrices. The result has 16 cells; each is one row times one column, added up (chapter 9).
3. Think of a CPU as one very fast calculator. It fills the cells one at a time, so 16 cells means waiting 16 turns.
4. But the cells don't need to wait for each other. Each cell can be computed as soon as it has its own row and its own column.
5. A GPU is thousands of simple calculators. Like 10,000 students each taking one cell, it fills every cell at once.
6. Say a calculator does 20 cells a second, a student 1. For 10,000 cells, the calculator takes 500 seconds; 10,000 students take 1!
7. So big AIs train on thousands of linked GPUs. Transformers compute all the words at once, a perfect match for GPUs (chapter 41).

Chapter summary

• Each result cell of a matrix product is its own 'one row · one column' calculation. No cell has to wait for another.
• CPU = a few fast calculators taking turns; GPU = thousands of simple calculators working at the same time.
• The more cells, the bigger the GPU's advantage. That's why AI is trained on GPUs.