Video Understanding
Recognising what's happening in video — where models score well on shuffled frames, which tells you what they actually learned.
Reviewed July 14, 2026Stable
When not to use it
- Trusting an action recognition benchmark. Shuffle the frames. If the score holds, it's scene classification.
- For temporal reasoning. "Picked up or put down" is the thing you wanted and the thing models fail.
- Processing every frame. 30 seconds is 900 images. Sample.
- Assuming a video model beats frame sampling. Sample a few frames into an image model first; it's a strong baseline.
Reach for something else instead
- Frame sampling + image model — the embarrassing baseline that's often competitive.
- Two-stream with optical flow — hand-deliver the motion, since the model won't learn it.
- Something-Something-style evaluation — if you want to know whether time is being used.
- VLM on sampled frames — ask a question about the video rather than classifying it.
Further reading
- Carreira & Zisserman (2017), Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset — I3D and the benchmark that drove the field.
- Goyal et al. (2017), The "Something Something" Video Database for Learning and Evaluating Visual Common Sense — built to defeat the scene shortcut; models do far worse.
- Feichtenhofer et al. (2019), SlowFast Networks for Video Recognition — two pathways, and a structural admission that semantics and motion are separate.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reading action recognition scores as temporal understanding. Shuffled frames barely hurt.
- Missing why two-stream works. If models learned motion from RGB, precomputed flow wouldn't help. It does.
- Processing video as many images. Absurd cost, and the sampling baseline is competitive anyway.
- Blaming the models. The benchmark permitted a shortcut and the objective was maximised. Nobody cheated.
At a glance
FieldComputer Vision
The findingshuffle the frames and scores barely drop
What that meansthe model recognises the scene, not the action
The diagnostictwo-stream works, which proves motion isn't learned from RGB
The honest benchmarkSomething-Something, where models do far worse
DifficultyIntermediate
Often compared with
Action recognition vs. temporal understanding — one is answered by a single frame of a basketball court; the other requires knowing whether he picked it up or put it down.
Where this sits
A destination. 12 concepts lead here, and nothing in the corpus depends on it.
9Levelsteps in
12Needs firstconcepts
0Opens upnothing further
1Areastays here
Learn these firstObject Tracking
Computed from the prerequisite graph, not assigned. How this works