AI Anatomy Class · Part 7 · Teaching GPT: pre-training

69. GPT-1 → 2 → 3

Q. From GPT-1 to GPT-3, how much did it grow in just two years?

1. GPT wasn't born all at once. GPT-1 came in 2018, GPT-2 in 2019, GPT-3 in 2020. It got bigger every year.
2. Let's say 100 million parameters (chapter 12's knobs) make one floor. GPT-1 has 117 million: a one-story house.
3. GPT-2 has 1.5 billion: a 15-story apartment building. About 13 times bigger in one year.
4. GPT-3 has 175 billion. How many floors? It won't fit on the screen. Let's step back.
5. A whopping 1,750 floors! About 117 times GPT-2, and about 1,500 times GPT-1. GPT-1 doesn't even show up as a dot.
6. You'd have to stack about 17 Empire State Buildings, 102 floors each, to reach it. Enormous!
7. The bigger GPT-3 could solve new tasks from just a few examples (chapter 71). Why does bigger mean smarter? Next chapter.

Chapter summary

• GPT-1 (2018) 117 million → GPT-2 (2019) 1.5 billion → GPT-3 (2020) 175 billion parameters.
• At 100 million = 1 floor: a 1-story house → a 15-story apartment building → a 1,750-floor tower.
• About 1,500 times bigger in two years. As the size grew, new abilities appeared (ch. 71).