AI Anatomy Class · Part 6 · Assembling the transformer

55. The thinking layer

Q. Attention let the words share information. What does each word do next?

1. With attention, the words mixed in each other's meanings. Now it's the block's second part: feed-forward.
2. Feed-forward is a small neural network each word goes through on its own (chapter 18). Every word uses the very same machine.
3. Inside, 4 numbers widen 4 times to 16, then shrink back to 4. In the wide layer, GELU bends the numbers (chapter 17).
4. Each neuron in the wide layer reacts to some pattern. Let's say this one lights up when it sees 'to' after a movement word.
5. What the lit-up neurons produce is added to the original vector. That's how knowledge gets attached to each word.
6. In GPT-3, 12,288 numbers widen to 49,152 and then come back to 12,288. The same 4 times.
7. About two-thirds of GPT-3's 175 billion numbers are in feed-forward. It's the warehouse of what the model learned.
8. Attention: words talk to each other. Feed-forward: each word thinks alone. Feed-forward is now filled in on the blueprint!

Chapter summary

• Feed-forward: a small neural network each word goes through on its own. Every word uses the same machine.
• It widens 4 times (bending with GELU), then narrows again. GPT-3: 12,288 → 49,152 → 12,288.
• Its result is added to the original vector, attaching knowledge. About two-thirds of the model's numbers live here.