AI Anatomy Class · Part 4 · Before the transformer

38. Looking at only the last few words

Q. How did AI find 'the chance of the next word' in the old days?

1. The easiest way to find a probability is to count. First we need a notebook full of sentences, called a corpus.
2. To count what follows 'I walk to', those three words must appear together. That happens just once, and 'We walk to' never does.
3. So here's a trick: look only at the one word right before. We just count what came after 'to'.
4. After 'to', 'school' came 3 times and 'work' once. So 'school' wins 3 times out of 4: 75%.
5. Count like this for every word and you get a pair-count table. Looking at only the last few words is called an n-gram.
6. Let's write a sentence using only this table. In 1948, Claude Shannon wrote English sentences this way too.
7. Every pair sounds natural, but the sentence is odd, right? Looking only one word back, it forgot the subject was a cat.
8. Why not look further back? Longer word strings are rarer in the corpus, so the count is often 0. Chapter 39's neural net fixes this.

Chapter summary

• n-gram: look at only the last few words, and count the next word in a corpus to get its probability.
• Each pair may sound natural, but the sentence easily goes off track because it forgets what came earlier.
• Looking further back doesn't work either: such long word strings are rarely in the corpus, so the count is 0.