AI Anatomy Class · Part 7 · Teaching GPT: pre-training

63. The internet is the textbook

Q. What books did ChatGPT actually study?

1. ChatGPT's textbook is the internet. News, encyclopedias, blogs, books, even code: a huge pile of text written by people.
2. But you can't use it as is. The same text is copied countless times, and ads, broken text and nasty words are mixed in.
3. So it goes through a text factory. Step one: remove duplicates. Of texts that are the same or nearly the same, keep just one.
4. Step two: the quality filter. It sifts out text with nothing to learn, like ads, random letters, or the same words repeated.
5. Step three: the harmful content filter. Out go insults, dangerous content, and personal info like phone numbers.
6. As an illustrative example: collect 100, and only 25 are left. Real ratios differ from dataset to dataset.
7. GPT-3 studied about 300 billion tokens of text chosen this way. As we saw in chapter 4, good data makes a good model.

Chapter summary

• ChatGPT's textbook = the internet's huge amount of text (news, encyclopedias, books, code…).
• It isn't used as is: remove duplicates → quality filter → harmful content filter.
• Filtering shrinks the pile, but good data makes a good model.