AI Anatomy Class · Part 11 · Graduation project

94. Training in the browser

Q. How does a model really learn? Watch it happen inside your own phone.

1. Time to train! At first, the mini GPT's 24,925 knobs are set at random, so it just spits out random pieces.
2. One training step: take 8 chunks of 24 tokens from the data, and at every position, guess the next token (chapter 64).
3. Wrong guesses give a loss (chapter 65). It starts near 5.86, the surprise of guessing at random among 349 kinds.
4. Backpropagation works out each knob's share of the blame (chapter 23), then nudges it to lower the loss. That's one step.
5. Go round 1,500 times and the loss keeps falling: below 2 at step 650, below 1 at step 1,100.
6. Its writing changes too: nonsense at step 100, real words by step 500, proverb-like lines by step 1,500.
7. But train long enough and it memorizes the data. Of 80 lines from the step-1,500 model, 27 were copied word for word (chapter 25).
8. Now press the train button in Try it. No server: the real math runs right inside your phone.

Chapter summary

• One step = guess the next token in 8 chunks → loss → backpropagation → turn the knobs.
• The loss falls from 5.86 (random guessing) to about 0.77 after 1,500 steps.
• Train long and it starts memorizing. Tell new lines apart from memorized ones.