Speculative Decoding
A small model guesses ahead and the big one checks in parallel — two to three times faster, with mathematically identical output. An actual free lunch.
When not to use it
- On highly creative or high-temperature text. The draft is wrong constantly and you pay for guesses you discard.
- Without a well-matched draft model. Below ~40% acceptance it costs more than it saves.
- When memory is the binding constraint. You're holding two models.
- On very short outputs. The overhead doesn't amortise.
Reach for something else instead
- Self-speculation (Medusa, EAGLE) — no separate draft model, which is the hard part.
- Quantization — smaller and faster, and it does trade quality.
- A smaller model — if you'd accept the quality drop, you didn't need this.
- Batching — if you're serving many requests, throughput may matter more than latency.
Sources & further reading
- Leviathan, Kalman & Matias (2022), Fast Inference from Transformers via Speculative Decoding — the method and the correctness proof.
- Chen et al. (2023), Accelerating Large Language Model Decoding with Speculative Sampling — independent concurrent work, at scale.
- Li et al. (2024), EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty — drafting in feature space; no separate model.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming it degrades quality. The output distribution is provably identical — that's the whole point.
- Using a mismatched draft model and getting a slowdown.
- Not measuring acceptance rate. It's the only number that tells you if this is working.
- Using it for creative generation, where the draft can't guess.
At a glance
FieldLanguage & LLMs
Ideasmall model guesses, big model verifies in parallel
Speedup2–3×, output distribution provably identical
Why it worksdecoding is memory-bound; the GPU was idle
The number to watchacceptance rate; >70% good, <40% bad
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Speculative decoding vs. quantization — both make inference faster; one provably changes nothing about the output, the other trades quality for it.