CLIP
Training on images and their captions until both live in one space — the model that connected vision to language, and the reason typing a prompt gets you a picture.
When not to use it
- For composition, counting or spatial relations. "Red cube on blue sphere" and its inverse embed close together.
- For negation. It has essentially no representation of "not."
- As an understanding model. It's a similarity model. Different thing.
- Training your own with small batches. Contrastive learning needs negatives. Use SigLIP.
Reach for something else instead
- SigLIP — better, no huge-batch requirement. The default now.
- A vision-language model — if you need reasoning about the image rather than matching.
- Supervised classification — fixed categories and labels? Still better.
- Captioning + text search — clumsier, more interpretable.
Sources & further reading
- Radford et al. (2021), Learning Transferable Visual Models From Natural Language Supervision — CLIP; zero-shot ImageNet from alt-text.
- Zhai et al. (2023), Sigmoid Loss for Language Image Pre-Training — SigLIP; kills the huge-batch requirement. Use this one.
- Liang et al. (2022), Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning — the shared space isn't shared.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Expecting compositional understanding. It was never in the loss.
- Fighting the prompt template. "A photo of a {class}" is free accuracy.
- Assuming image and text embeddings mix. The modality gap says they don't.
- Missing the bias loop — it learned internet associations and now filters the next generation's training data.
At a glance
FieldComputer Vision
Year2021
Trained on400M image-caption pairs, zero labels
The shockzero-shot ImageNet competitive with a ResNet-50 trained on it
What it powersevery text-to-image model's conditioning
The lessonfree noisy supervision at scale beat curated labels
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
CLIP vs. supervised classification — one learns from free noisy captions and generalises to categories nobody labelled; the other needs annotation and still wins on the categories you have.