AI Anatomy Class · Part 9 · The moment an answer is made

83. The AI's memory limit

Q. In a long chat, ChatGPT sometimes forgets what you said at the start. Why?

1. As we saw in chapter 77, a chat is one long text to the model. But there's a limit to how much it can read at once.
2. That limit is called the context window. GPT-3 could see up to 2,048 tokens. Only text inside the window is used to predict.
3. As the chat grows, every new message pushes the oldest one out of the window.
4. Now ask for your name. It's outside the window, so the model can't see it. It may say it doesn't know, or invent one (chapter 84).
5. A bigger window is costly. Attention scores every pair of pieces (chapter 47), so double the length means four times the table.
6. When writing, each new piece needs the Key and Value of every earlier piece (chapter 46). Recomputing them each time is wasteful.
7. The KV cache reuses Key·Value notes already computed. Earlier pieces never look ahead (chapter 51), so their values don't change.
8. But the notes pile up as the chat grows and take up memory. In a long chat, it helps to repeat the important parts.

Chapter summary

• Context window: how many tokens the model can see at once. GPT-3: 2,048 tokens.
• Past the window, the oldest words get pushed out first, and the model can no longer see them.
• KV cache: keep notes of the Keys and Values already computed and reuse them to answer faster.