AI Anatomy Class · Part 6 · Assembling the transformer

59. Encoder vs decoder

Q. The 'GPT' in ChatGPT and Google's 'BERT' are both transformers. So what's the difference?

1. The first Transformer (2017) was a translator: an encoder reads the original, and a decoder writes the translation (chapter 44).
2. In 2018 came models that used just one half: BERT, a stack of encoders only, and GPT, a stack of decoders only.
3. BERT sees the whole sentence from both sides. It practices by hiding a word in the middle and guessing it from its neighbors.
4. GPT sees only the words before. With chapter 51's mask hiding the future, it practices guessing the next word.
5. In the attention tables, every cell is open for BERT. For GPT, only the diagonal and the triangle below it are open.
6. So BERT is strong at understanding sentences, and GPT is strong at writing text one word at a time.
7. ChatGPT is a GPT made only of decoder blocks. The block we've assembled since chapter 54 is exactly this decoder block.

Chapter summary

• The original Transformer (2017) was a translator made of an encoder + a decoder.
• BERT = encoders only: sees both sides and fills in blanks. Strong at understanding sentences.
• GPT = decoders only: sees only the words before (the mask) and guesses the next one. Strong at writing on. ChatGPT is this kind.