AI Anatomy Class · Part 5 · Mastering attention

44. Attention Is All You Need

Q. The paper that became ChatGPT's backbone claimed 'attention is all you need'. Why?

1. In 2017, eight researchers who worked together at Google published a paper. Its title: 'Attention Is All You Need'.
2. Here are the eight authors. A footnote in the paper says the order of the names is random: they all contributed equally.
3. Back then, RNNs (ch. 40) were the stars of translation. They read one word at a time, so each cell had to wait for the one before.
4. Chapter 43's attention was a helper for RNNs. This paper flipped the question: what if we removed the RNN and kept only attention?
5. Then nothing has to wait. All the words look at each other at the same time, so a GPU can compute them all at once (ch. 41).
6. The model built this way is the transformer. It beat the best translation scores of its time and trained in far less time.
7. GPT was built from this transformer too: the T in GPT stands for Transformer. From GPT-1 in 2018 all the way to ChatGPT.
8. In this Part we'll take apart attention, the heart of this paper, piece by piece. First, chapter 45: words looking at each other.

Chapter summary

• 2017, 'Attention Is All You Need': a paper by eight researchers working at Google.
• Removing the RNN and keeping only attention let every word be computed at the same time.
• This design is the transformer. The T in GPT stands for Transformer.