AI Anatomy Class · Part 6 · Assembling the transformer

58. Stacking blocks

Q. One block already works. So why stack 96 of them?

1. What goes into a block comes out in the same shape. So, like Lego, you can keep snapping more blocks on below it.
2. GPT-1 has 12 layers, GPT-2 has 48, and GPT-3 has 96. They stacked the very same block design over and over.
3. The shape is the same, but the numbers inside differ from layer to layer. So each layer does a slightly different job.
4. Shallow layers mostly handle how a word looks and its neighbors; middle layers handle roles and relations in the sentence.
5. Deeper layers handle the whole sentence's meaning and what comes next. Information piles up on the same 'to', layer by layer.
6. If we peek at the next-word guess after each layer, the probability of 'school' rises the deeper we go.
7. One GPT-3 block layer has about 1.8 billion numbers. All 96 add up to about 174 billion, almost all of the 175 billion.
8. Stacking this deep works thanks to the highway and normalization. Next chapter: two ways to use the block.

Chapter summary

• A block's output has the same shape as its input, so blocks stack. GPT-1: 12 layers, GPT-2: 48, GPT-3: 96.
• The design is the same, but each layer has its own numbers, so shallow and deep layers do different jobs.
• The deeper you go, the more context and next-word information a word vector carries. Almost all of GPT-3's numbers are in the blocks.