AI Anatomy Class · Part 4 · Before the transformer

39. The neural language model

Q. How can an AI guess the next word in a sentence it has never seen before?

1. An n-gram may have counted 'the cat' lots of times, but if it has never seen 'the dog', it has no idea what comes next. The count is 0.
2. In 2003, Bengio's team came up with a new way: a neural network that guesses the next word, the neural language model.
3. First, every word gets a vector of numbers: chapter 33's embedding. Words used in similar ways sit close together.
4. The vectors of the last three words are joined into one row and fed in. After Part 2's neuron layers, each candidate word gets a score.
5. Softmax (ch. 14) turns the scores into probabilities: school 65.7%, the 24.2%, work 8.9%, pizza 1.2%.
6. The dog vector is close to the cat vector, so the network gives similar answers. It can guess even in a sentence it has never seen.
7. Training adjusts the network's weights and the word vectors together. Ten years later, Word2Vec (ch. 34) focused only on the vectors.
8. One limit remains: it's still a fixed window that sees only the last few words. How do we see the whole sentence? Chapter 40.

Chapter summary

• 2003 neural language model: word vectors (embeddings) + a neural network predict the next word.
• Similar words = nearby vectors, so it makes good guesses even for sentences it has never seen.
• Training polishes the word vectors too. Limit: it's still a window that sees only the last few words.