AI Anatomy Class · Part 9 · The moment an answer is made

82. Narrowing the candidates

Q. If the wheel has a 'pizza' slice, won't 'pizza' really come up someday?

1. Sampling has a trap. There are tens of thousands of candidates, so all the tiny bars together make a fat tail.
2. Even if the tail is only 3%, in a 100-piece answer the chance that an odd word sneaks in somewhere is 95%.
3. So we cut the tail. First, Top-k: keep only the top k by probability. With k = 3, three are left.
4. Share the remaining 85% out again so it adds up to 1: school 61.2%, the 24.7%, work 14.1%. Now spin the wheel.
5. Second, Top-p: add up the probabilities from the top, stop the moment you pass p, say 90%, and cut the rest.
6. Top-k always keeps 3, but Top-p adapts: it keeps 1 when the model is sure and 6 when it's unsure.
7. So: divide the scores by temperature, softmax them, cut the tail, and spin the wheel. That's how each piece is picked.

Chapter summary

• There are tens of thousands of candidates, so even a thin tail can add up and let an odd word slip in.
• Top-k keeps the top k. Top-p keeps adding from the top until the total passes p.
• After cutting, share the rest out again so it adds up to 1, then spin the wheel.