AI Anatomy Class · Part 4 · Before the transformer

42. How a translator is built

Q. Two languages with different word orders and lengths. How does an AI translate between them?

1. 'I walk to school' in Spanish is 'Voy a la escuela caminando'. The order and the word count differ, so you can't swap word by word.
2. In 2014, Sutskever's team joined two RNNs: an encoder that reads and a decoder that writes. This is called seq2seq.
3. The encoder reads the English one word at a time, rewriting its note. The last note holds the meaning of the whole sentence.
4. The decoder takes that note and writes Spanish one word at a time. Each word it writes goes back in as input, just like chapter 2.
5. But the note has a fixed size. Four words or forty, the whole sentence must be squeezed into one note of the same size.
6. That's the bottleneck: all the meaning must pass through one narrow neck. So the longer the sentence, the worse the translation got.
7. What if the decoder could look back at the original every time it writes a word? That idea is chapter 43's attention.

Chapter summary

• seq2seq (2014): an encoder RNN reads, and a decoder RNN writes one word at a time.
• The encoder hands over the whole sentence as one note (vector) of a fixed size.
• Bottleneck: the longer the sentence, the less fits in that one note, so the translation gets worse.