Multimodal AI
Models that handle more than one kind of input — text and images, sometimes audio and video — in a single shared representation.
When not to use it
- For precise extraction from dense documents. Specialised OCR is more accurate and far cheaper on scanned text at scale. Multimodal models are strong at understanding a page and weak at reading every character off it.
- When exact numbers matter and you can't verify them. A model misreading a figure produces a plausible number, not an error message.
- On a budget, at volume. Images cost many tokens each. A pipeline that would be pennies with OCR can be substantial with a frontier model.
Reach for something else instead
- Dedicated OCR for text extraction — decades of engineering aimed at exactly this.
- A classifier if the task is "which of these five categories is this image."
- CLIP-style embeddings when you need image-text matching or search rather than reasoning or description.
Sources & further reading
- Radford et al. (2021), Learning Transferable Visual Models From Natural Language Supervision — CLIP, the shared image-text space that most of this rests on.
- Alayrac et al. (2022), Flamingo: a Visual Language Model for Few-Shot Learning — bridging a vision encoder into a language model.
- Liu et al. (2023), Visual Instruction Tuning — LLaVA, and the recipe that made open vision-language models practical.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Sending a full page and asking about small print. The model downscales; the detail is gone before it reads anything. Crop first.
- Trusting counts and spatial relations. "How many people are in this photo" and "what's to the left of the chair" are known weak spots.
- Ignoring image token cost until the bill. Resolution is a budget lever, and most people find that out late.
At a glance
FieldGenerative AI
Core ideaseveral input types, one shared space
Strong atdescribing, reasoning about images
Weak atprecise reading, counting, spatial detail
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Multimodal AI vs. OCR — understanding a page in context vs. transcribing every character precisely.