AI Anatomy Class · Part 4 · Before the transformer

40. RNN: passing notes in a relay

Q. Sentences come in every length. How does an AI remember what came before while it reads?

1. Chapter 39's network saw only three words, but sentences come in every length. New idea: read one word at a time and pass a note along!
2. Each cell gets one word. It takes the note from the cell before, rewrites it after reading its word, and hands it to the next cell.
3. 'Yesterday' goes in the 'when' slot, 'at school' in 'where', and 'I' in 'who'. What was there before is copied a little fainter.
4. Every cell uses the same rule: multiply the old note by 0.8, add the current word, then squash the result to between −1 and 1.
5. Only the last note is used to guess the next word. 'Yesterday' is still on it, so 'went' gets 70%. The note is called the hidden state.
6. Change just the first word to 'Tomorrow', and now 'will go' gets 70%. The first word rode the notes all the way to the end.
7. Unrolled it's a relay; rolled up it's one machine going round in a loop. That's why it's called a recurrent neural network, or RNN.

Chapter summary

• RNN: reads one word at a time, rewrites a note (the hidden state), and passes it to the next cell.
• Every cell uses the same rule (the same weights), so sentences can be any length.
• The last note guesses the next word. Even the first word rides the notes to the end.