AI Anatomy Class · Part 7 · Teaching GPT: pre-training

67. A smarter way downhill

Q. Why does a ball that only follows the slope get lost?

1. In chapter 21, the ball rolled down by looking only at the slope under its feet. But a real loss landscape is bumpy like this.
2. A ball that only follows the slope stops in a small dip. At the bottom of the dip the slope is 0, so there's nowhere to go.
3. So we give it momentum. It remembers how it was rolling, and that push carries it over the little hill to the deeper valley.
4. Problem two is step size. Each knob has its own slope: the steep one bounces back and forth, the gentle one crawls like a snail.
5. So each knob gets its own step size. A knob that has been swinging wildly takes small steps; a sluggish one takes big strides.
6. Momentum plus automatic step sizes is a method called Adam. It came out in 2014, and GPT is trained with Adam too.
7. The real math is more complex, but these two ideas are the heart of it. Step ④ of chapter 66's loop just got smarter.

Chapter summary

• Following only the slope, the ball gets stuck in a small dip or zigzags on the steep side.
• Momentum: remembering how it was rolling carries the ball over small hills.
• Adam = momentum + an automatic step size for each knob. It's used to train GPT, too.