AI Anatomy Class · Part 7 · Teaching GPT: pre-training

70. Bigger is smarter

Q. Make a model 10 times bigger: how much better does it get? Can we know in advance?

1. Does a bigger model really get smarter? In 2020, researchers built many models of different sizes and compared their loss.
2. Let's plot it on a normal scale. The small models crowd together at the left edge, and after that it looks almost flat.
3. Now let's change the horizontal ruler. Instead of adding 1 with each step, each step multiplies by 10.
4. Now the dots line up in a straight line. In this illustrative example, every 10 times bigger cuts the loss by 0.5.
5. A ruler that grows 10 times per step is a log scale. Like surprise in chapter 65, it counts how many times you multiplied.
6. More data or more computing gave the same straight line. This rule is called the scaling laws.
7. Extend the line, and you can predict the loss of an even bigger model. That's why people dared to build big models like GPT-3.
8. In 2022, the Chinchilla study said: don't just grow the model; grow the data too, about 20 tokens of text per parameter.

Chapter summary

• Change the horizontal ruler to steps of ×10 (a log scale), and the loss drops in a straight line.
• Grow the size, data and compute, and the loss drops predictably = scaling laws (2020).
• Extending the line predicts big models. The data has to grow too (Chinchilla, 2022).