Edge AI
Running models on the device instead of a server — for privacy, latency and cost, against a memory wall that doesn't move.
When not to use it
- When you need a frontier model. The memory wall is real and it isn't moving much.
- For sustained inference on battery. It drains and thermally throttles. A 30-second model isn't shipped.
- Assuming one build works everywhere. Chip fragmentation is most of the engineering cost.
- For cost reasons alone. At low volume, an API is cheaper than the engineering.
Reach for something else instead
- Hybrid — small on-device, escalate the hard cases. What everyone actually ships.
- Server inference — batching, and the economics that come with it.
- Distillation — a small model that imitates a big one, often better than training small.
- Task-specific adapters — swap a LoRA over a shared base rather than shipping models.
Sources & further reading
- Howard et al. (2017), MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications — designed for the constraint, not shrunk into it.
- Jacob et al. (2018), Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference — the practical quantization foundation.
- Alizadeh et al. (2024), LLM in a flash: Efficient Large Language Model Inference with Limited Memory — paging weights from flash; the memory wall attacked directly.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Optimising compute. Memory bandwidth is the wall.
- Forgetting batch size is permanently 1. Every server-side efficiency trick is unavailable.
- Ignoring thermal limits. Sustained inference throttles the device.
- Choosing edge for cost at low volume. The engineering exceeds the API bill.
At a glance
FieldTools & Ecosystem
The wallmemory bandwidth, not compute
Batch sizepermanently 1; every server efficiency trick unavailable
The default4-bit quantization
What everyone shipshybrid, with routing
The durable reasonprivacy is architectural, not a promise
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Edge vs. server inference — one gives you a privacy guarantee that needs no trust and a batch size of 1 forever; the other gives you batching economics and a promise.