AI Anatomy Class · Part 1 · AI speaks in numbers

14. Scores into probabilities: softmax

Q. A model's scores can be negative, and they add up to anything. How do we turn them into probabilities?

1. The model gives each candidate next word a score. A score can be 3, or even a negative number like −1.
2. But probabilities can't be below 0, and together they must add up to 1. Right now one is negative and the total is 5.
3. Step one: raise e to the power of each score. e is a number, about 2.718. This makes every value positive.
4. As a bonus, big scores get much bigger. A score of 3 was 1.5 times a 2; after the change, it's about 2.7 times.
5. Step two: add up all the new values. The total is 30.56. Now divide each value by this total.
6. Out come probabilities: school 65.7%, the 24.2%, work 8.9%, pizza 1.2%. Together, exactly 100%.
7. These two steps together are called softmax. You'll meet it again with attention in chapter 45, so remember it.

Chapter summary

• Scores (logits) can be negative, and they add up to anything.
• ① Raising e to each score makes everything positive, and big scores get even bigger.
• ② Dividing by the total gives probabilities. These two steps are softmax.