AI Anatomy Class · Part 2 · How machines learn

17. Bend to survive: activation functions

Q. Tacos are best when they're spicy, but not too spicy. Can a neuron figure that out?

1. Here are 13 trucks with the same price, placed by spiciness and line. Circles are the hits our friends picked; crosses are misses.
2. The chapter 16 neuron splits the plane with one straight line. But however you move or turn the line, at least two are always wrong.
3. A neuron only multiplies and adds, so its boundary is always straight. Chain neurons together: line plus line is still a line.
4. So we bend the neuron's output once. ReLU turns negatives into 0 and lets positives through. This is called an activation function.
5. With bent lines, we can make a V-shaped boundary that cleanly splits hits from misses. Chapter 18 shows how it's built.
6. GPT uses GELU, a smoothed-out ReLU. It's rounded near 0, and on the negative side it dips just a little before returning to 0.
7. Without activation functions, any stack is just one straight line. Bending is what lets a network learn complex shapes.

Chapter summary

• A single neuron (multiply and add) always has a straight boundary. Line + line = line.
• Activation function = something that bends the output. ReLU: negative → 0, positive → unchanged.
• GPT uses GELU, a smoothed ReLU. Bending is what makes complex shapes learnable.