AI Anatomy Class · Part 6 · Assembling the transformer

56. The information highway

Q. After 96 layers, wouldn't the information in the original sentence be wiped out?

1. Ever played the telephone game? The more people a message passes through, the stranger it gets.
2. Stacking layers deep does the same thing. If every layer replaces the whole vector, the original information fades.
3. So the block doesn't swap the vector out. It adds each part's result to the original vector: x + f(x).
4. The original x stays as it is, and the part adds only a small fix. So the original information isn't erased.
5. In the picture, the original information runs straight down the highway, and the parts add results from the side roads.
6. It helps training too. The 'fix this much' signal rides the highway all the way back to the first layer (chapter 23).
7. This path that adds results to the original is called a residual connection. Each part only handles the 'difference to fix'.

Chapter summary

• If every layer replaces the whole vector, the original information fades, like in the telephone game.
• Residual connection: output = x + f(x). The original rides the highway; each part only adds the difference to fix.
• The fix signal also rides the highway back to the early layers, so the model can be stacked deep.