AI Anatomy Class · Part 0 · ChatGPT, who are you?

2. An AI that writes one word at a time

Q. Why does ChatGPT's answer show up bit by bit, as if someone were typing?

1. Ask a question and the answer appears piece by piece. That's no special effect: it really is made one piece at a time.
2. Look inside: the whole text so far, your question plus the answer so far, goes into the model. One next piece comes out.
3. The piece joins the end, and the longer text goes back in. Writing by rereading its own output like this is called autoregressive.
4. So when does it stop? When a special piece that means "the end" is picked. It's called the end token.
5. 9 pieces means 9 runs of the model; 100 pieces, 100 runs. So longer answers take longer.
6. And once a piece is added, it's never erased. If something earlier was wrong, the fix has to be written after it.
7. The same question gets a slightly different answer each time because pieces are drawn by probability. More in chapter 81.

Chapter summary

• Put the whole text in, and one next piece comes out.
• Add that piece and put the text back in, until the end token comes out.
• The model runs once per piece, and added pieces are never erased.