AI Anatomy Class · Part 2 · How machines learn

22. Choosing the step size: learning rate

Q. Walking down a mountain, how big should each step be?

1. In chapter 21, one step was learning rate × slope. The learning rate is the step size: how big a stride each step takes.
2. If the learning rate is too small (0.01), it's slow as a turtle. After five steps, it's still halfway down the slope.
3. Just right (0.1), and it reaches the bottom in a few steps: the very steps from chapter 21.
4. Too big (0.3), and it overshoots the bottom and bounces to the other side. Each bounce goes higher, so the loss grows.
5. Plot the loss after each step and the difference jumps out: a slow slide, a quick drop, and a shoot upward.
6. So the learning rate is chosen carefully. GPT-3 raised it a little at first, then slowly lowered it as training went on.
7. Too small is a turtle; too big bounces away. The right step size matters. Smarter steps come in chapter 67.

Chapter summary

• Learning rate = the size of one step. New knob = knob − learning rate × slope.
• Too small and it crawls like a turtle; too big and it overshoots the bottom and bounces away.
• The right learning rate reaches the bottom fast. It's often lowered during training.