AI Anatomy Class · Part 2 · How machines learn

21. Gradient descent

Q. To shrink the loss, which way should you turn a knob, and by how much?

1. Take just Truck A and turn one knob of the chapter 16 neuron: the line weight w. Right now, w is 1 and the loss is 1.
2. Try many values of w and plot the loss, and you get a valley. We want to find its lowest point.
3. But in real life you can't see this picture: there are countless knobs. It's like a foggy mountain where you only see your feet.
4. You can feel the slope under your feet, though. Nudge w up by just 0.01, and the loss drops from 1 to 0.96, about 0.04 less.
5. The slope is how much the loss changes for a small nudge: −0.04 ÷ 0.01 = about −4. It's negative, so raising w goes downhill.
6. So take one step against the slope! New w = 1 − 0.1 × (−4) = 1.4. The 0.1 is the step size (chapter 22, learning rate).
7. Repeat this and the ball rolls to the bottom: w is 1.5, the guess is 4.5, the loss is 0. This is gradient descent.
8. With two knobs it's a bowl. With 175 billion it's an unimaginable mountain, but finding each knob's slope works the same way.

Chapter summary

• Slope = how much the loss changes when you nudge a knob a little.
• Step a little against the slope: new knob = knob − learning rate × slope.
• Repeat, and you go down to the bottom of the loss valley. That's gradient descent.