AI Anatomy Class · Part 2 · How machines learn

18. Stacking layers

Q. A problem one neuron can't solve: can a group of neurons solve it?

1. How do we make chapter 17's V-shaped boundary? One neuron can only draw a straight line, so we group neurons into a layer.
2. The first layer has two neurons. One computes spicy − 3, the other 3 − spicy, and both use ReLU to turn negatives into 0.
3. Add the two values and you get "how far from level 3". Levels 1 and 5 both give 2; levels 2 and 4 both give 1.
4. On the plane, it's like folding the paper in half along the level 3 line. The points on the left flip over onto the right.
5. On the folded paper, the V straightens into a line. Now a single neuron in the next layer splits them with one straight line.
6. Stack more layers and you can fold again and again. Two folds make a W; many folds make very complex boundaries.
7. A network stacked in layers like this is called a multilayer perceptron (MLP). GPT-3 uses 49,152 of these neurons per layer.

Chapter summary

• Several neurons side by side = a layer. Several layers chained = a multilayer perceptron (MLP).
• Two ReLU neurons can "fold" the space. Folding turns a V into a straight line.
• The deeper the network, the more it can fold, making more complex boundaries.