AI Anatomy Class · Part 2 · How machines learn

27. Part 2 boss battle

Q. Put together everything so far: can we train a neural network from scratch?

1. It's the Part 2 boss battle! Chapter 17's truck map now has 60 trucks. This time, the network finds the boundary by itself.
2. 2 inputs → 8 ReLU neurons → 1 output (chapter 18). Its 33 knobs start out random. A score over 0.5 means a hit.
3. Run the training loop: forward pass, loss, backpropagation to share the blame, and gradient descent to fix (chapters 19–23).
4. Minibatches of 10, for 300 epochs (chapter 24). As the loss drops, the boundary grows into a V, and accuracy goes from 23% to 100%.
5. We check with 30 new trucks set aside (chapter 25). All 30 are right: it didn't memorize; it learned the V-shaped rule.
6. GPT learns exactly the same way, only with 175 billion knobs instead of 33, and 300 billion tokens instead of 60 trucks.
7. Part 2's learning tools are complete! In Part 3, we learn how to turn text into numbers so we can feed it to a network.

Chapter summary

• Training loop: forward pass → loss → backpropagation → fix by gradient descent, repeated in minibatches, epoch after epoch.
• People choose the number of neurons, the learning rate and the epochs. Training finds the knobs (weights, biases).
• Real skill shows on validation data. GPT learns the same way, just far bigger.