AI Anatomy Class · Part 4 · Before the transformer

43. The birth of attention

Q. What if a translator could glance back at the original while it writes? How much better would AI translation get?

1. To fix chapter 42's bottleneck, Bahdanau's team had a new idea in 2014: don't keep only the last note, keep every note!
2. When the decoder writes 'Voy', it scans all the saved notes and shines a spotlight where it's needed right now: 'I' gets 70%.
3. For the next word, 'a', the spotlight skips over 'walk' and lands on 'to'. A different word order is no problem.
4. 'la' and 'escuela' light up 'school', and 'caminando' jumps back to 'walk'. Each word it writes looks somewhere different.
5. How much light each note gets comes from relevance scores turned into shares by softmax (ch. 14). The notes are mixed in those shares.
6. Draw a table of where it looked and you see the lines cross. Matching up word pairs like this is called alignment.
7. Let's wrap up Part 4: counting (n-gram), neural nets and word vectors, the note relay (RNN), and now attention.
8. Then, in 2017, someone asked: what if we dropped the note relay and built everything from attention alone? See you in Part 5.

Chapter summary

• Attention (2014): each time the decoder writes a word, it looks again at every note from the original.
• Relevance scores → softmax → shares. The notes are mixed in those shares.
• Different word orders and long sentences are OK. In 2017: attention alone (Part 5)!