AI Anatomy Class · Part 4 · Before the transformer

41. The limits of RNNs

Q. If the note relay is so good, why doesn't ChatGPT use an RNN?

1. A long sentence: 'Yesterday after class and a long walk with my dog, I ___'. The answer should be 'went', because of 'Yesterday'.
2. Each time the note is passed on, 'Yesterday' gets a little fainter. After 11 cells, you can barely see it.
3. So the RNN, looking only at the last note, picks 'go'. It missed a link between far-apart words: a long-term dependency.
4. Training has the same problem. Blame passed backward (ch. 23) shrinks at every cell: 10 cells back, only 0.11 of it is left.
5. In 1997, the LSTM gave the note a storage box with gates, so important things fade less. But in very long texts, they still fade.
6. The second problem is waiting in line. Cell 2 can't start until cell 1's note arrives. The other GPUs sit idle, waiting their turn.
7. GPUs are great at doing many calculations at once (ch. 68). But with 1,000 words, an RNN has to take 1,000 turns, one after another.
8. So RNN notes fade, and RNNs wait in line. The transformer fixes both (Part 5). But first, chapter 42: how a translator is built.

Chapter summary

• Limit ①: the note fades at every cell, so links between far-apart words (long-term dependencies) get lost.
• The 1997 LSTM remembers longer with a storage box and gates, but very long texts still fade.
• Limit ②: each cell must wait for the one before, so GPUs wait in line instead of working at once.