Speculative Decoding
A small model guesses ahead and the big one checks in parallel — two to three times faster, with mathematically identical output. An actual free lunch.
Reviewed July 13, 2026Stable
When not to use it
- On highly creative or high-temperature text. The draft is wrong constantly and you pay for guesses you discard.
- Without a well-matched draft model. Below ~40% acceptance it costs more than it saves.
- When memory is the binding constraint. You're holding two models.
- On very short outputs. The overhead doesn't amortise.
Reach for something else instead
- Self-speculation (Medusa, EAGLE) — no separate draft model, which is the hard part.
- Quantization — smaller and faster, and it does trade quality.
- A smaller model — if you'd accept the quality drop, you didn't need this.
- Batching — if you're serving many requests, throughput may matter more than latency.
Read more on the blog
- Speculative decoding: faster LLM generation, same outputThere is a way to make a large language model generate text two to four times faster while producing output that is mathematically identical to the slow way. It sounds impossible, but it works, and it is now standard in production serving. The trick is to let a small model guess ahead and have the big model check the guesses in parallel.
- Why AI gives different answers: sampling and temperatureAsk a chatbot the same question twice and you often get two different answers. That is not a glitch. A language model does not choose the next word; it rolls weighted dice over thousands of options, and a setting called temperature controls how loaded those dice are. Here is how the last step of generation actually works.
Further reading
- Leviathan, Kalman & Matias (2022), Fast Inference from Transformers via Speculative Decoding — the method and the correctness proof.
- Chen et al. (2023), Accelerating Large Language Model Decoding with Speculative Sampling — independent concurrent work, at scale.
- Li et al. (2024), EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty — drafting in feature space; no separate model.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming it degrades quality. The output distribution is provably identical — that's the whole point.
- Using a mismatched draft model and getting a slowdown.
- Not measuring acceptance rate. It's the only number that tells you if this is working.
- Using it for creative generation, where the draft can't guess.
At a glance
FieldLanguage & LLMs
Ideasmall model guesses, big model verifies in parallel
Speedup2–3×, output distribution provably identical
Why it worksdecoding is memory-bound; the GPU was idle
The number to watchacceptance rate; >70% good, <40% bad
DifficultyAdvanced
Often compared with
Speculative decoding vs. quantization — both make inference faster; one provably changes nothing about the output, the other trades quality for it.
Where this sits
A destination. 25 concepts lead here, and nothing in the corpus depends on it.
12Levelsteps in
25Needs firstconcepts
0Opens upnothing further
1Areastays here
Computed from the prerequisite graph, not assigned. How this works