Inference API
Renting a model by the request — how nearly everyone actually uses AI, and the dependency that comes with it.
Reviewed July 11, 2026Stable
When not to use it
- When the data legally can't leave. This is a contract question, and it's the one genuine reason self-hosting wins regardless of arithmetic.
- At very high sustained volume, where the arithmetic does eventually flip — but check, rather than assume it already has.
- When you need a fixed model forever. Providers deprecate, and a product depending on exact behaviour is exposed to someone else's roadmap.
Reach for something else instead
- Self-hosted open-weight models when data residency or volume genuinely justifies the operations.
- A smaller model — often the task never needed a frontier one, and nobody tested.
- No model — if a rule solves it, a rule is faster, cheaper, and correct.
Further reading
- Pope et al. (2022), Efficiently Scaling Transformer Inference — what the provider is doing to make your request cheap.
- Yu et al. (2022), Orca: A Distributed Serving System for Transformer-Based Generative Models — continuous batching, the technique behind modern serving throughput.
- Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention — vLLM, and why serving got cheaper.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Calling the API without retry and backoff, so a rate limit becomes an outage.
- Treating it as a reliable local function. It's a network call to a busy service and it will fail.
- Logging every prompt for debugging and creating an undeclared store of whatever users typed.
At a glance
FieldTools & Ecosystem
Core idearent inference per request
You buysomeone else's operations
You give updata control, model permanence
Mitigationabstract the provider
DifficultyBeginner
Often compared with
Inference API vs. self-hosting — renting operations vs. owning them, and the arithmetic is rarely where people guess.
Where this sits
A destination. 6 concepts lead here, and nothing in the corpus depends on it.
4Levelsteps in
6Needs firstconcepts
0Opens upnothing further
1Areastays here
Learn these firstModel Serving
Computed from the prerequisite graph, not assigned. How this works