Video Understanding
Recognising what's happening in video — where models score well on shuffled frames, which tells you what they actually learned.
When not to use it
- Trusting an action recognition benchmark. Shuffle the frames. If the score holds, it's scene classification.
- For temporal reasoning. "Picked up or put down" is the thing you wanted and the thing models fail.
- Processing every frame. 30 seconds is 900 images. Sample.
- Assuming a video model beats frame sampling. Sample a few frames into an image model first; it's a strong baseline.
Reach for something else instead
- Frame sampling + image model — the embarrassing baseline that's often competitive.
- Two-stream with optical flow — hand-deliver the motion, since the model won't learn it.
- Something-Something-style evaluation — if you want to know whether time is being used.
- VLM on sampled frames — ask a question about the video rather than classifying it.
Sources & further reading
- Carreira & Zisserman (2017), Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset — I3D and the benchmark that drove the field.
- Goyal et al. (2017), The "Something Something" Video Database for Learning and Evaluating Visual Common Sense — built to defeat the scene shortcut; models do far worse.
- Feichtenhofer et al. (2019), SlowFast Networks for Video Recognition — two pathways, and a structural admission that semantics and motion are separate.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reading action recognition scores as temporal understanding. Shuffled frames barely hurt.
- Missing why two-stream works. If models learned motion from RGB, precomputed flow wouldn't help. It does.
- Processing video as many images. Absurd cost, and the sampling baseline is competitive anyway.
- Blaming the models. The benchmark permitted a shortcut and the objective was maximised. Nobody cheated.
At a glance
FieldComputer Vision
The findingshuffle the frames and scores barely drop
What that meansthe model recognises the scene, not the action
The diagnostictwo-stream works, which proves motion isn't learned from RGB
The honest benchmarkSomething-Something, where models do far worse
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Action recognition vs. temporal understanding — one is answered by a single frame of a basketball court; the other requires knowing whether he picked it up or put it down.