Test-Time Compute
Spending more compute when the model answers rather than when it trains — the scaling axis the field found after the first one got expensive.
Reviewed July 16, 2026Moving
When not to use it
- Tasks with no verifier and no majority to take. Extra samples give you more text and no way to choose, which is spending without buying.
- Latency-critical paths. This axis converts compute into time by construction.
- Anything where a cheaper model already saturates the task. Test-time compute is a multiplier on a gap; with no gap there is nothing to multiply.
Reach for something else instead
- A better base model is the training-time lever, and for high-volume products it is usually the cheaper one, amortised.
- Retrieval fixes the knowledge failures people often try to fix with more thinking. The model wasn't reasoning badly; it didn't have the fact.
- A verifier plus a small model frequently beats a large model thinking hard, because the check contributes more than the thinking.
Sources & further reading
- Snell et al. (2024), Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters — the compute-allocation result.
- Brown et al. (2024), Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — coverage rises log-linearly with samples; verification is the constraint.
- Wang et al. (2023), Self-Consistency Improves Chain of Thought Reasoning in Language Models — the simplest method that works.
- Lightman et al. (2023), Let's Verify Step by Step — process supervision, the verifier side of the trade.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming more samples means better answers. Coverage rises; selection doesn't come free, and without a verifier you're picking the most plausible-looking, which is not the same thing.
- Spending the same budget on every question. The optimal allocation depends on difficulty — uniform spend wastes most of it on questions that were easy.
- Treating it as a substitute for training-time scale. They're different axes with different economics, and the right split depends on your query volume, not on which is fashionable.
At a glance
FieldFoundations
The tradecompute at answer time vs. training time
Bottleneckverification, not generation
Made famous byreasoning models, 2024–25
DifficultyIntermediate