How AI generates video: from noise to motion

Two years ago, AI video was a curiosity: a few seconds of warping, dreamlike footage where faces melted and objects drifted through each other. By 2026 the same prompt produces something startling, a coherent clip of ten to twenty-five seconds, often with synchronised sound, where a character keeps the same face across the whole shot and a thrown ball roughly obeys gravity. Text-to-video became one of the fastest-moving frontiers in AI, and it is worth understanding both because it is impressive and because it stretches the ideas behind image generation to their limit.

This piece follows on from how AI generates images, and it assumes that foundation: video generation is, at its heart, image generation extended into time. What that extension requires, why the added dimension of time is what makes the problem hard, the architecture that now dominates the field, and where the technology still falls short: those are the threads worth pulling. The short version of the difficulty is that making one frame look good was mostly solved by diffusion; making a thousand frames agree with each other is the real work.

The new problem is time

A video is a sequence of images, called frames, played fast enough that the eye reads motion. So a first guess at AI video would be: generate a bunch of images and play them in order. This fails immediately, and the reason it fails is the key to the whole subject.

If you generate each frame independently, nothing connects them. The character's shirt is blue in frame one, teal in frame two, green in frame three. The coffee cup on the table moves position, or vanishes, or multiplies. Motion looks like a flickering slideshow of related but inconsistent pictures rather than a continuous scene. This property, or its absence, is called temporal consistency: the requirement that objects keep their identity, appearance, and plausible motion across frames. It is the single hardest thing about video generation, and it is what separates a cheap model (objects subtly morph, colours shift, motion has an uncanny fluidity) from a top-tier one (objects hold stable identity and motion looks physically plausible).

So the real task is not "generate frames" but "generate frames that are aware of each other," each one consistent with the ones around it, all of them describing a single evolving scene. The model has to handle space (what each frame looks like) and time (how the frames relate) together. Almost every design decision in video generation follows from this one demand.

It is still diffusion, extended into time

The good news is that the core engine is one you already understand from image generation: diffusion. The model is trained by taking real videos, adding noise until they become static, and learning to reverse that process, removing noise step by step. To generate, it starts from random noise and denoises repeatedly until a video emerges, with a text prompt steering each step toward matching your words. The denoise-from-noise principle is unchanged.

The change is what the model denoises. Instead of a single 2D image, it works on a block of frames at once, a chunk of video with both spatial dimensions (height and width) and a time dimension (frame order). And critically, the denoising network sees the whole block together, so when it decides how to clean up frame twenty, it can look at frames nineteen and twenty-one. That shared view across time is what lets it keep things consistent: the shirt stays blue because the model is denoising all the frames as one connected object, not each in isolation. Temporal consistency is not bolted on afterward; it comes from generating the frames jointly rather than separately.

The dominant design: diffusion transformers and spacetime patches

The architecture that made modern video generation work, used by the leading 2026 systems, has a name worth knowing: the diffusion transformer, or DiT. It combines the diffusion process above with the transformer architecture, and the way it does so reuses an idea from language models in a clever form.

Recall that a transformer processes a sequence of tokens using attention, letting every token consider every other. A diffusion transformer treats video as a sequence too. It chops the block of video into small patches across both space and time, "spacetime patches," and treats those patches much like a language model treats word tokens. Each patch is a small piece of the scene at a particular place and moment. Attention then lets every patch attend to every other patch, across space and across time. A patch in frame twenty can directly attend to the corresponding patch in frame one, which is exactly the mechanism that enforces long-range consistency: the model can connect distant frames the way a language transformer connects distant words. This is why diffusion transformers handle temporal coherence better than the older convolutional designs they replaced, and why they scale predictably as you add compute, the same scaling property that made transformers dominate language.

The reason this runs at all is the same compression trick from image generation. Denoising raw video pixels would be impossibly expensive, so the video is first encoded into a compressed latent space using a variational autoencoder, a spatiotemporal grid far smaller than the raw pixels. The whole diffusion process runs in that compact space, and a decoder expands the final result back to full-resolution video. Without this, minute-long high-resolution generation would not be feasible on any reasonable hardware.

The 2026 state of the art

A few developments define where the technology sits now, and they are worth naming because they show both how far it has come and what still constrains it.

Native audio. The newest models generate sound with the video rather than adding it later. A model like Veo 3 applies diffusion jointly to visual and audio latents, so at each denoising step the attention mechanism works over a unified sequence of both picture and sound, producing synchronised dialogue, effects, and ambient noise in a single pass. Video and its audio are generated as one object, which is why the lip movements and the speech line up.

Length and cost. The attention that gives DiT its consistency also carries the transformer's central weakness: its cost grows quadratically with sequence length, and a video is a very long sequence of patches. This is why generated clips are still measured in seconds rather than minutes, and why longer durations get expensive fast. Length, not per-frame quality, is now the binding constraint for most systems.

The long-video question. To push past the length limit, a smaller branch of research generates video autoregressively, producing frames or chunks in sequence, each conditioned on what came before, the way a language model writes word by word. This has a theoretical advantage for long, coherent video, but it can accumulate errors over time (small mistakes compound frame after frame), and in 2026 it has not displaced the parallel DiT approach, which still wins on quality per training dollar. As with image generation, the field has not converged, and hybrids of the two are an active direction.

Video as world simulator. The most ambitious framing is that a model good enough to generate physically plausible video has implicitly learned something about how the world works, a world model. To render a convincing bouncing ball or pouring liquid, the model must capture regularities of physics and object permanence. Whether this makes video models a path toward AI that understands the physical world, or whether they are sophisticated appearance-matchers that only approximate physics, is one of the more interesting open debates the technology has opened up.

What it still gets wrong

An accurate picture has to include the failure modes, because they are specific and visible once you know to look. Physics is approximated, not understood: objects can pass through each other, liquids behave oddly, and complex interactions (a hand grasping an object, a crowd moving) still break. Long-range coherence remains a common weakness across the whole model class, so identity can drift over a longer clip even in strong models. Fine detail flickers, especially text and faces at a distance. And the compute cost keeps most output short. These are not reasons to dismiss the technology, which is already good enough for real production use in marketing and pre-production, but they mark the current edge between what looks convincing and what falls apart.

The short version

AI generates video with the same diffusion process used for images, extended so that the model denoises a whole block of frames together rather than one at a time, which is what keeps objects consistent across the sequence. The dominant design, the diffusion transformer, chops video into spacetime patches and uses attention to let every patch relate to every other across space and time, which is what enforces long-range temporal coherence and lets the approach scale. The whole process runs in a compressed latent space to make it affordable. The hard part throughout is time: keeping a thousand frames agreeing with each other, obeying rough physics, and holding identities stable, all of which the best 2026 models do impressively but imperfectly.

The idea to hold onto is that video generation is not about making frames look good, which diffusion mostly solved, but about making frames agree, which is why the whole architecture is built to let distant moments in the clip see and constrain each other. Making a picture was the easy half. Making a picture move without falling apart is the frontier.

Common questions

How does AI video generation work? Most AI video generators use diffusion, the same technique behind AI image generation, extended into time. The model is trained by adding noise to real videos until they become static, then learning to reverse the process. To generate, it starts from random noise and denoises step by step into a video, steered by a text prompt. The key difference from image generation is that it denoises a whole block of frames together rather than one at a time, so the frames stay consistent with each other and objects keep their identity across the clip.

What is a diffusion transformer (DiT)? A diffusion transformer is the architecture behind most leading 2026 video generators, including Sora and Veo. It combines diffusion (denoising noise into content) with the transformer architecture. It chops video into small patches across space and time, called spacetime patches, and treats them like the tokens a language model processes, using attention to let every patch relate to every other across both space and time. This is what gives it strong temporal consistency and lets it scale predictably with more compute.

What is temporal consistency in AI video? Temporal consistency is the requirement that objects keep their identity, appearance, and plausible motion across the frames of a video. Without it, a character's clothing shifts colour between frames, objects move or vanish, and motion looks like a flickering slideshow rather than a continuous scene. It is the hardest part of video generation and the main thing separating cheap models (where objects subtly morph) from top-tier ones (where motion looks physically plausible and identities stay stable). Modern models achieve it by generating all the frames jointly rather than independently.

Why are AI-generated videos so short? Because the attention mechanism that keeps the frames consistent has a cost that grows quadratically with the length of the video, treated as a sequence of patches. Doubling the length roughly quadruples the computation, so longer videos become expensive fast. This is why most 2026 clips are measured in seconds rather than minutes. Per-frame quality is largely solved; the binding constraint now is length and its compute cost, which is an active area of research including autoregressive approaches that generate video in chunks.

How is AI video generation different from image generation? Both use diffusion, denoising random noise into content guided by a prompt. The difference is the added dimension of time. Image generation produces one still picture; video generation must produce many frames that agree with each other, keeping objects consistent and motion plausible across the sequence. This is handled by denoising a block of frames together and by using attention across both space and time, and it introduces challenges image generation does not face, chiefly temporal consistency, approximate physics, and much higher compute cost.

Do AI video models understand physics? Not in a deep sense. To generate convincing motion, a model must capture regularities like object permanence and rough physical behaviour, which is why some researchers describe strong video models as implicit world models that have learned something about how the world works. But the physics is approximated from patterns in training footage, not computed, which is why objects can still pass through each other and liquids behave oddly. Whether video models are a path toward genuine physical understanding or sophisticated appearance-matchers is an open debate.

Can AI generate video with sound? Yes. The newest models generate audio together with the video rather than adding it afterward. A model like Veo 3 applies the diffusion process jointly to visual and audio data in a compressed latent space, so at each step the attention mechanism operates over a combined sequence of both, producing synchronised dialogue, sound effects, and ambient noise in a single generation pass. Generating the picture and its sound as one object is what makes elements like lip movement and speech line up.