Vision Transformer
Cut an image into patches, treat them as words, run a transformer — which works, and only if you have enough data.
When not to use it
- From scratch on a small dataset. It has to learn what a CNN knows for free. Use a CNN or a pretrained ViT.
- When compute is tight at high resolution. Attention is quadratic in patch count; halving patch size quadruples the cost.
- On dense prediction, in plain form. Detection and segmentation want multi-scale structure. Hierarchical variants exist for a reason.
- Assuming the architecture is the win. ConvNeXt suggests much of it was the training recipe.
Reach for something else instead
- CNN / ConvNeXt — modernised, competitive, better with less data.
- Swin and hierarchical ViTs — locality reintroduced; better for detection and segmentation.
- Hybrids — convolutional stem, transformer body. Often the practical best.
- A pretrained backbone — what you'll actually do, which makes the argument moot.
Sources & further reading
- Dosovitskiy et al. (2020), An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — the paper; note "at Scale" is in the title and gets dropped from the citation.
- Touvron et al. (2021), Training data-efficient image transformers & distillation through attention — DeiT; the data requirement was partly the recipe.
- Liu et al. (2022), A ConvNet for the 2020s — ConvNeXt; modernise a CNN and it matches. Awkward for the strong reading.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Dropping "at scale" from the result. It's in the title of the paper.
- Training a ViT from scratch on 20k images and concluding transformers don't work for vision.
- Changing input resolution without interpolating the position embeddings.
- Reading ViT's win as architectural. DeiT and ConvNeXt both complicate that.
At a glance
FieldComputer Vision
Idea16×16 patches as tokens, then a plain transformer
What it gives uplocality and translation equivariance
The tradeassumptions help when data is scarce, cap you when it isn't
The caveat"at Scale," which gets dropped
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
ViT vs. CNN — one knows nothing about images and learns everything; the other is born knowing locality and translation equivariance. Which wins is decided by how much data you have.