Text-to-Video
Generating video from a description — not image generation with more frames, because the hard part is that things must stay themselves.
When not to use it
- When you need a specific shot. Plausible clip ≠ your clip, and direction is the expensive part of film.
- For anything longer than a few seconds. Coherence degrades and no setting fixes it.
- When physical accuracy matters. Contacts, liquids and collisions are almost right, which is worse than obviously wrong.
- On undisclosed training data, commercially. Video provenance is murkier than images and the outputs are worth more.
Reach for something else instead
- Image-to-video — start from a frame you chose. Far more controllable and how most real work is done.
- Stock footage — licensed, clear, immediate.
- Traditional VFX — for anything requiring specificity, still faster than fighting a generator.
- Animation tools — if you need control over motion, tools that give control are the answer.
Sources & further reading
- Ho et al. (2022), Video Diffusion Models — extending diffusion across time; where temporal consistency gets addressed directly.
- Blattmann et al. (2023), Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models — latent video diffusion; the compute answer.
- Brooks et al. (2024), Video generation models as world simulators — the spacetime-patch framing and the world-model claim, from the people making the claim.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming it's image generation with more frames. Temporal consistency is a different and harder problem.
- Prompting for motion and expecting precision. Scene description works; motion description barely does.
- Judging on a curated demo reel. The failures are the informative part and they're not in the reel.
- Reading physics competence as a world model. The failure cases argue against it, and that argument is live.
At a glance
FieldGenerative AI
The hard parttemporal consistency, not image quality
Breaks firstobject permanence, physics, hands
Honest lengtha few seconds
More controllableimage-to-video
Live argumentworld model or appearance model
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Text-to-video vs. image-to-video — one invents everything including the thing you wanted; the other animates a frame you already approved. The second is a tool.