Speculative decoding: faster LLM generation, same output

Here is a claim that should make you suspicious: there is a technique that makes a large language model generate text two to four times faster, costs no quality at all, and produces output that is provably identical to what the model would have produced slowly. Speedups usually come with trade-offs, so "faster and exactly the same" sounds like it breaks some conservation law. It does not. The technique is speculative decoding, it is now standard in production inference systems, and understanding how it pulls off "free" speed is a small lesson in where the waste in LLM inference actually hides.

This piece explains the idea from the ground up: why generating text leaves the hardware half-idle, how a small "draft" model and a large "verifier" model combine to fill that idle capacity, why the result is guaranteed to match the large model exactly rather than approximate it, and when the trick delivers a big speedup versus barely any. It builds directly on why inference is memory-bound, and it is one of the more satisfying pieces of engineering in modern AI serving, because it buys real speed without giving anything up.

The waste it exploits: idle compute during generation

To see why speculative decoding is possible, recall the central fact about how inference works. Generating text happens one token at a time, sequentially, and each step is memory-bound: to produce a single token the model must read its entire set of weights from memory, but it does only a small amount of arithmetic with them. The compute units finish their little job quickly and then sit idle, waiting on the next batch of weights to arrive from memory. The bottleneck is data movement, not calculation.

This leaves an opening. During each generation step the GPU's compute capacity is mostly unused. What if you could do more arithmetic per weight-load, checking several possible tokens at once instead of computing just one, without paying much extra, because the compute was idle anyway? That is exactly the gap speculative decoding drives through. It converts the wasted parallel compute of the memory-bound decode phase into extra tokens per step.

The core idea: draft, then verify

Speculative decoding uses two models: a large, accurate target model (the one you actually want output from) and a small, fast draft model (a cheaper model, often a smaller member of the same family). The two play a guess-and-check game.

The loop goes like this. First, the small draft model quickly generates a guess of the next several tokens, say the next four, running sequentially but on a model small enough that this is cheap. Then, and this is the key move, the large target model verifies all four guessed tokens in a single parallel forward pass. Because the transformer can process a whole sequence of tokens at once (the same parallelism that makes the prefill phase fast), checking "would I have produced these four tokens?" costs the target model roughly the same as generating one token would have. The target then accepts the longest correct prefix of the guess, the tokens that match what it would have generated, and rejects the rest. Wherever the first mismatch occurs, the target substitutes its own correct token there, and the next round begins from that point.

The win is arithmetic. In a single expensive target-model pass, which normally yields exactly one token, you now confirm as many tokens as the draft got right, plus one correction. If the draft guessed all four correctly, you produced five tokens for the price of one target pass. Even if it only got two right, you got three tokens for one pass. You are still paying for the draft model's work, but the draft is small and fast, so as long as it guesses well often enough, the net result is several tokens per expensive step instead of one. The idle compute of the memory-bound target model is now doing useful verification.

Why the output is identical, not approximate

The property that makes speculative decoding remarkable, and that separates it from ordinary shortcuts, is that it is lossless. The output is not "close to" what the target model would have produced; it is drawn from exactly the same distribution, as if the draft model were never involved. This is guaranteed by the acceptance rule, and the logic is worth following because it is what makes the whole thing trustworthy.

When the target verifies a drafted token, it does not just check for a match; it uses a carefully designed acceptance-and-correction procedure. If the drafted token is one the target would plausibly have generated, it is accepted. If it is not, it is rejected, and the replacement token is sampled from the target's own distribution, adjusted to account for what was already proposed. Worked through the mathematics of sampling, this rejection-and-resampling scheme has a clean guarantee: the final sequence of tokens follows precisely the same probability distribution as the target model generating alone. The draft model only ever affects speed, never which tokens ultimately come out. A wrong guess by the draft costs a little wasted work but cannot corrupt the result, because every token that survives has passed the target's own bar. This is why speculative decoding can be turned on in production without changing model behaviour at all: it is a pure acceleration, exact by construction.

What decides the speedup: acceptance rate

Speculative decoding is not equally fast in all cases, and the single number that governs it is the acceptance rate: how often the draft model's guesses match what the target would have produced. The higher the acceptance rate, the more drafted tokens survive each verification, and the bigger the speedup. In practice, useful acceleration wants an acceptance rate above roughly 80 percent and a draft length of about four to eight tokens.

Acceptance rate depends on how predictable the text is. For repetitive or highly structured output, boilerplate code, formatted data, formulaic prose, the next tokens are easy to guess, so a small draft model agrees with the large one most of the time and the speedup is large. For creative, high-entropy, surprising text, where even the large model's next token is highly uncertain, the draft guesses wrong more often, acceptance falls, and the speedup shrinks. This is the honest limitation: speculative decoding accelerates the predictable parts of generation most, and the unpredictable parts least. It also relies on the draft model being both fast (or it adds too much overhead) and accurate (or its guesses get rejected), which is the tension every variant tries to balance.

The variants: where the draft comes from

The classic setup uses a separate small model as the drafter, but that is only one way to produce guesses, and much of the research since has focused on better sources of drafts. It is worth knowing the main families, because they show the design space.

Self-speculative methods avoid a second model entirely by using the target model to draft against itself, for instance by skipping some of its own layers to produce a fast, rough guess that the full model then verifies. This eliminates the memory cost of a separate draft model, though the speedup it can reach is more limited. Medusa attaches a few extra lightweight prediction heads directly onto the target model, each trained to predict a token a few positions ahead, so the target effectively drafts several future tokens for itself in parallel. EAGLE and its successors draft at the level of the model's internal features rather than raw tokens, reusing the target's own top-layer representations to produce unusually accurate guesses, which pushes acceptance rates and speedups higher (reported gains range widely, often around three to six times, depending on model and workload). The through-line across all of them is the same tension: a bigger, smarter drafter gets more guesses accepted but costs more to run, while a smaller drafter is cheaper but gets rejected more often, and the art is finding the balance for a given model and task.

Why it matters now

Speculative decoding matters because it attacks inference cost from an angle orthogonal to the others. Quantization reduces the bytes per weight; mixture of experts reduces the parameters computed per token; speculative decoding reduces the number of expensive sequential steps. They stack. And because it is exact, it carries none of the quality risk that quantization does at low precision, which is why it has been adopted natively in the major serving frameworks and turned on by default in many deployments.

It is especially valuable for the workloads that generate long outputs, which is increasingly where the cost lives. Reasoning models that produce long chains of thought spend most of their time in the sequential decode phase, generating token after token, which is exactly the phase speculative decoding accelerates. As models are asked to think longer and write more, a technique that multiplies decode throughput without touching output quality becomes not a nice-to-have but a core part of making them affordable to run.

The short version

Speculative decoding speeds up text generation by having a small, fast draft model guess the next several tokens and a large, accurate target model verify all of them in a single parallel pass. Because verifying several tokens costs the target about the same as generating one, and because the decode phase leaves the GPU's compute idle anyway, each expensive target pass now yields several tokens instead of one whenever the draft guesses well. A rejection-and-resampling acceptance rule guarantees the output is drawn from exactly the same distribution as the target model alone, so the speedup is lossless: same output, faster. The gain depends on the acceptance rate, which is high for predictable text and lower for creative text, and variants differ mainly in where the draft tokens come from.

The idea to hold onto is that generation is slow because it is sequential and leaves compute idle, and speculative decoding fills that idle compute by guessing several tokens ahead and verifying them at once, buying real speed with a mathematical guarantee that the output never changes. It is one of the rare optimisations with no catch on quality, and the reason is that it does not approximate the model at all. It just stops making the model wait its turn one token at a time.

Common questions

What is speculative decoding? Speculative decoding is a technique that accelerates text generation from a large language model by using a small, fast "draft" model to guess several upcoming tokens and a large, accurate "target" model to verify them all in a single parallel pass. Because verifying several tokens costs roughly the same as generating one, each expensive pass of the large model can produce several tokens instead of one whenever the draft's guesses are correct. Crucially, it produces output mathematically identical to the large model running alone, so it speeds up generation without any loss of quality.

How does speculative decoding make output faster without changing it? It exploits two facts. First, generating a token is memory-bound and leaves the GPU's compute mostly idle, so extra parallel work is nearly free. Second, a transformer can verify many tokens at once in a single pass for about the cost of generating one. The draft model proposes several tokens; the target verifies them in parallel and accepts the ones it agrees with. A careful acceptance-and-resampling rule guarantees the accepted tokens follow exactly the target's own distribution, so the draft affects only speed, never which tokens come out.

Is speculative decoding lossless? Yes. Unlike approximations such as aggressive quantization, speculative decoding is exact: the output is drawn from precisely the same probability distribution as the target model generating on its own. This is guaranteed by the rejection-and-resampling acceptance rule, which rejects any drafted token the target would not have produced and replaces it with a token sampled from the target's own adjusted distribution. A wrong guess by the draft model wastes a little work but cannot corrupt the result, which is why speculative decoding can be enabled in production with no change to model behaviour.

What is the acceptance rate in speculative decoding? The acceptance rate is how often the draft model's guessed tokens match what the target model would have generated, and it is the main factor determining the speedup. A higher acceptance rate means more drafted tokens survive each verification, so more tokens are produced per expensive target pass. Useful speedups generally want an acceptance rate above about 80 percent with a draft length of four to eight tokens. Acceptance is high for predictable, structured text like code and low for creative, high-entropy text where the next token is highly uncertain.

When does speculative decoding not help much? When the text being generated is unpredictable. For creative writing or other high-entropy output, even the large model is quite uncertain about the next token, so the small draft model guesses wrong often, the acceptance rate falls, and few drafted tokens survive each verification. In that case the overhead of running the draft model is not repaid by enough accepted tokens, and the speedup shrinks toward nothing. Speculative decoding accelerates predictable, structured generation (like code and formatted output) far more than surprising, creative generation.

What are Medusa and EAGLE? They are variants that change where the draft tokens come from to avoid running a full second model. Medusa attaches a few extra lightweight prediction heads to the target model itself, each predicting a token several positions ahead, so the model drafts for itself in parallel. EAGLE drafts at the level of the model's internal features rather than raw tokens, reusing the target's own representations to produce highly accurate guesses and higher speedups. Both aim to raise the acceptance rate or cut the drafting overhead compared with using a separate standalone draft model.

Does speculative decoding work with quantization and other optimizations? Yes, and that is part of why it is valuable. It attacks inference cost from a different angle than the others: quantization reduces bytes per weight, mixture of experts reduces parameters computed per token, and speculative decoding reduces the number of sequential decode steps. They stack together. Because speculative decoding is exact and changes nothing about the output, it adds no quality risk, which is why it is supported natively in major serving frameworks and especially useful for reasoning models that spend most of their time generating long sequences.