AI Anatomy Class · Part 0 · ChatGPT, who are you?

5. The blueprint we will build

Q. If you opened up the machine called ChatGPT, what would you find inside?

1. To sum up so far: ChatGPT is a machine that guesses the next piece. Now it's time to open that machine up.
2. Up front, the tokenizer cuts text into pieces, the embedding turns them into numbers, and position info tells each piece's place.
3. Next, transformer blocks: attention lets pieces look at each other, and feed-forward adds thinking. Dozens are stacked.
4. At the end, softmax gives the next piece's probabilities, and one is picked and added. Then back to the top.
5. But the numbers inside start out a mess. Practicing next-piece guessing on a huge amount of text fixes them: that's pre-training.
6. Finally, it learns to chat from model answers and people's praise. Then it becomes ChatGPT.
7. We'll fill in this blueprint one Part at a time. So far we've filled in "pick and append" and the training data. Ready to start?

Chapter summary

• Tokenizer → embedding + position → [attention + feed-forward] × dozens of layers → softmax → pick and append.
• Numbers that start out a mess get fixed by pre-training.
• Learn to chat from model answers and people's ratings, and you get ChatGPT.