AI Anatomy Class · Part 6 · Assembling the transformer

54. The transformer block blueprint

Q. The 'transformer block' sits in the middle of the chapter 5 blueprint. What's inside if you open it?

1. Let's zoom in on the middle of the chapter 5 blueprint, the transformer block. GPT is many of these blocks stacked up.
2. One vector goes into the block for each word. The same number of vectors, the same size, comes back out.
3. The first part is attention. The words look at each other and mix their meanings. We learned it in chapter 45.
4. The second is feed-forward. It adds a 'thought' to each word separately. We'll open it up in the next chapter.
5. Each part's result is added to the original vector. The straight road the original information travels is called the highway.
6. Right before each part, normalization evens out the size of the numbers. It's the method you learned in chapter 15.
7. In short: normalize → attention → add, then normalize → feed-forward → add. That's one layer of the block.
8. What goes in comes out in the same shape, so blocks can be stacked again and again. GPT-3 has 96 layers of this block.

Chapter summary

• One block layer = normalize → attention → add, then normalize → feed-forward → add.
• Attention mixes the words together; feed-forward lets each word think. Each result is added to the original vector.
• What goes in comes out in the same shape, so blocks can be stacked. GPT-3 has 96 layers.