Vision-Language Model (VLM)
A model that takes images and text in the same input and reasons across both — the architecture behind every AI that can look at a screenshot.
Reviewed July 16, 2026Stable
When not to use it
- High-volume, narrow, stable vision tasks. A small trained classifier is cheaper, faster, calibrated, and won't hallucinate a label that isn't in your taxonomy.
- Precise measurement, counting, or spatial relations. This is the documented weak spot and fluent output hides it.
- Anything needing determinism or an audit trail. The same image can produce different answers, and the answer is prose, not a score.
Reach for something else instead
- A purpose-trained classifier for a fixed label set — better on every metric except flexibility.
- OCR plus a text model is often more accurate and far cheaper for documents, because dedicated OCR beats a VLM at reading small text.
- Classical CV — edges, contours, template matching — still wins for measurement and inspection, where the answer must be a number.
Sources & further reading
- Radford et al. (2021), Learning Transferable Visual Models From Natural Language Supervision — CLIP; the shared image-text embedding space.
- Alayrac et al. (2022), Flamingo: a Visual Language Model for Few-Shot Learning — bridging a frozen vision encoder into a frozen LLM.
- Liu et al. (2023), Visual Instruction Tuning — LLaVA; the encoder + projection + LLM recipe most open VLMs follow.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming it can see what you can see. Fine text and dense tables often fall below the effective patch resolution, and the model reads them wrong rather than declining.
- Budgeting images like text. A single image can cost more tokens than the prompt around it, and image-heavy pipelines blow through context windows and budgets simultaneously.
- Trusting published benchmark scores. Multimodal benchmarks are contaminated and many questions are answerable without the image at all — build your own eval or you're buying a number, not a capability.
At a glance
FieldComputer Vision
Typical buildvision encoder + projection + LLM
Made practical byCLIP (2021), LLaVA (2023)
Weak atcounting, precise spatial relations
DifficultyIntermediate