Inference API
Renting a model by the request — how nearly everyone actually uses AI, and the dependency that comes with it.
When not to use it
- When the data legally can't leave. This is a contract question, and it's the one genuine reason self-hosting wins regardless of arithmetic.
- At very high sustained volume, where the arithmetic does eventually flip — but check, rather than assume it already has.
- When you need a fixed model forever. Providers deprecate, and a product depending on exact behaviour is exposed to someone else's roadmap.
Reach for something else instead
- Self-hosted open-weight models when data residency or volume genuinely justifies the operations.
- A smaller model — often the task never needed a frontier one, and nobody tested.
- No model — if a rule solves it, a rule is faster, cheaper, and correct.
Sources & further reading
- Pope et al. (2022), Efficiently Scaling Transformer Inference — what the provider is doing to make your request cheap.
- Yu et al. (2022), Orca: A Distributed Serving System for Transformer-Based Generative Models — continuous batching, the technique behind modern serving throughput.
- Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention — vLLM, and why serving got cheaper.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Calling the API without retry and backoff, so a rate limit becomes an outage.
- Treating it as a reliable local function. It's a network call to a busy service and it will fail.
- Logging every prompt for debugging and creating an undeclared store of whatever users typed.
At a glance
FieldTools & Ecosystem
Core idearent inference per request
You buysomeone else's operations
You give updata control, model permanence
Mitigationabstract the provider
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Inference API vs. self-hosting — renting operations vs. owning them, and the arithmetic is rarely where people guess.