AI Anatomy Class · Part 10 · The world after ChatGPT

86. AI with eyes

Q. How can an AI that only read text look at a photo and describe it?

1. AI today can tell you what's in a photo. But a model is a machine that reads tokens (chapter 29). How do you put a photo in?
2. A photo is pixel numbers (chapter 6). Even a small 64×64 photo has 4,096 pixels. One token per pixel makes the line far too long.
3. So we cut the photo like a checkerboard into 16×16-pixel pieces called patches. A 64×64 photo becomes 4×4 = 16 pieces.
4. Each patch is 16×16 pixels × 3 colors (red, green, blue) = 768 numbers. Matrix multiplication (chapter 11) makes them one vector.
5. Line the patches up in reading order and add position information (chapter 36). Now it looks just like a line of word tokens.
6. A 2020 paper put this idea right in its title: 'An Image is Worth 16x16 Words'. The model is called ViT.
7. Many photo-chatting AIs read image tokens, then word tokens, in one line. Handling text and images together is called multimodal.

Chapter summary

• Cut a photo into 16×16-pixel pieces (patches), and each piece becomes one token.
• A matrix multiplication turns a patch's 768 numbers into a vector, and position information is added.
• Read image tokens and word tokens in one line, and you get a multimodal AI (ViT, 2020).