AI Anatomy Class · Part 2 · How machines learn

24. Splitting the workbook

Q. A workbook with millions of problems: fix after every problem, or solve them all and fix once?

1. Tuning the line knob to 1.5 makes A spot on, but B goes way off. The loss over all five trucks grows from 0.7 to 1.95.
2. So we must look at the whole workbook. But with millions of problems, solving them all before every fix is too slow.
3. So we solve a few at a time: solve a bundle, grade it, fix the knobs, then on to the next bundle. Each bundle is a minibatch.
4. Going through the whole workbook once, start to finish, is 1 epoch. Usually we repeat many epochs.
5. Compare over 180 problems: one at a time drops fast but wobbles; all at once fixes only 3 times. Bundles of 10 are smooth.
6. Computers calculate a whole bundle at once, so it's fast (chapter 68). GPT-3 packed about 3.2 million tokens into each bundle.
7. Split into minibatches, repeat for many epochs. But solving the same workbook too many times causes a problem (chapter 25).

Chapter summary

• Fix based on just one problem and the others go off. You have to look at the whole workbook.
• Minibatch: solve, grade and fix a few problems at a time. Epoch: one lap of the workbook.
• One at a time wobbles; all at once is slow. A medium bundle is fast and steadier.