AI Anatomy Class · Part 2 · How machines learn

23. Backpropagation: sharing the blame

Q. With billions of knobs, when the answer is wrong, which knobs do you fix, and by how much?

1. Chapter 19's network guessed 3.5 for Truck A; the real rating is 4.5, so the loss is 1. Which of the 11 knobs is to blame?
2. You could nudge each knob a little, as in chapter 21. But GPT-3 has 175 billion knobs, so that would never end.
3. So we hand out blame backward from the end. Raise the rating by 0.01 and the loss drops by 0.02: the rating's blame is −2.
4. The rating was made from popular × 1.5, so popular's blame is −2 × 1.5 = −3. Blame flows backward along the lines.
5. The line weight is multiplied by line 2 to make popular, so its blame is −3 × 2 = −6. The multiplications link up like a chain.
6. The "too spicy" neuron was off. An off neuron can't change the result, so its blame is 0. Those knobs stay as they are.
7. One sweep backward gives every knob's blame at once. This is backpropagation.
8. Nudge each knob against its blame: the guess becomes 4.2 and the loss falls from 1 to 0.09. This caught on in 1986 (chapter 3).

Chapter summary

• Backpropagation: start at the loss and go backward, multiplying blame along the lines.
• A knob's blame = how much the loss changes if you nudge it. Behind an off neuron, it's 0.
• One backward sweep gives every knob's blame; then nudge each a little the other way.