Optical Flow
Estimating the motion of every pixel between two frames — and there's a proof you fundamentally can't, from looking at any one part of the image.
When not to use it
- Through occlusion. A covered pixel has no correspondence. There is no correct answer; every method invents one.
- On textureless regions. The aperture problem at maximum. Confident flow there is a prior, not a measurement.
- On water, smoke or glass. Brightness constancy is simply false.
- For large fast motion, with classical methods. The object moved further than the search window.
Reach for something else instead
- Feature tracking (Lucas-Kanade) — sparse, fast, and often all you need.
- Learned video features — skip explicit flow; let the model handle motion implicitly.
- Block matching — what video codecs actually do. Crude and fast.
- Depth + ego-motion — if you want 3D motion, estimate that instead.
Sources & further reading
- Horn & Schunck (1981), Determining Optical Flow — brightness constancy plus smoothness; the formulation that still frames it.
- Teed & Deng (2020), RAFT: Recurrent All-Pairs Field Transforms for Optical Flow — correlation volume plus iterative refinement; the modern answer.
- Butler et al. (2012), A Naturalistic Open Source Movie for Optical Flow Evaluation — Sintel; and note the field trains on synthetic data because real ground truth is nearly unobtainable.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Expecting correct flow at motion boundaries. Smoothness is false exactly there, and smoothness is what made it solvable.
- Treating flow in a blank region as a measurement. There's no information; you're reading the regulariser.
- Forgetting the models trained on synthetic data. Real ground truth is nearly unobtainable.
- Thinking a better architecture solves the aperture problem. The information isn't in the data.
At a glance
FieldComputer Vision
What it givesa motion vector per pixel
The aperture problemone equation, two unknowns; you cannot solve it from the data
What makes it solvablean assumption you impose (smoothness), and that's where the errors live
Modern answerRAFT: correlation volume + iterative refinement
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Optical flow vs. a measurement — one of them tells you what's there. Flow in a textureless region is the model's prior, not information from the image.