Image Captioning
Describing an image in words — declared solved on benchmarks a decade ago, and the benchmarks were measuring the wrong thing.
When not to use it
- A dedicated captioning model. Use a VLM and ask for what you want. It's dramatically better.
- BLEU or CIDEr as your metric. They reward the modal caption and never check whether it's true.
- Generic captions for accessibility. "A group of people" helps nobody. The context is the requirement.
- Trusting the objects mentioned. Object hallucination is documented — language priors insert what usually co-occurs.
Reach for something else instead
- Vision-language models — ask a specific question, get a specific answer.
- CLIP — for retrieval and ranking, if you don't need generation.
- Human captions — for accessibility that matters, still the standard.
- CHAIR / SPICE — if you must have a metric, at least use one that checks something real.
Sources & further reading
- Vinyals et al. (2015), Show and Tell: A Neural Image Caption Generator — the CNN-to-LSTM template.
- Xu et al. (2015), Show, Attend and Tell — attention, and visualisable evidence it was looking at the right thing.
- Rohrbach et al. (2018), Object Hallucination in Image Captioning — CHAIR; models describe objects that aren't there, from language priors.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reporting CIDEr. Models trained to maximise it produce worse captions by human judgement, and this has been measured repeatedly.
- Treating benchmark saturation as task completion. The benchmark was solved a decade ago; the task wasn't.
- Assuming mentioned objects are present. That's object hallucination, and it's the same failure as in language models.
- Shipping generic alt text. The one application that matters is the one where blandness fails hardest.
At a glance
FieldComputer Vision
The 2015 templateCNN encoder, LSTM decoder
What went wrongoverlap metrics reward the modal caption, so models learned blandness
The failure modeobject hallucination; language priors insert what usually co-occurs
The application that mattersaccessibility, where generic captions are useless
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Captioning model vs. asking a VLM — one gives you the statistically safest sentence about the image; the other answers the question you actually had.