Guardrails
The checks around a model that decide what it's allowed to receive, say, and do — the part that stops a demo becoming an incident.
When not to use it
- As a substitute for scope. If the model shouldn't be able to do something, remove the capability rather than filtering the request. Filters fail; missing permissions don't.
- As a claim of safety. Passing your own checks means your checks passed. It says nothing about what you didn't think to check.
- Where friction outweighs risk. Aggressive filtering on a low-stakes internal tool costs you users and buys very little.
Reach for something else instead
- Reduced permissions — read-only access, spending caps, no destructive tools. The strongest control available, and it's not AI.
- Human approval for consequential actions. Slower per action, cheaper than one incident.
- Structured output — if the model can only emit a value from a fixed list, most output filtering becomes unnecessary.
Sources & further reading
- Greshake et al. (2023), Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — the paper that made this concrete.
- Perez & Ribeiro (2022), Ignore Previous Prompt: Attack Techniques For Language Models — the original demonstration, still clarifying.
- Bai et al. (2022), Constitutional AI — model-based supervision, and its limits.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Asking the model not to do the thing. Instructions in the prompt are advisory, and an injected instruction has equal standing.
- Only guarding output. Injection arrives on the input side, often through retrieved documents rather than the user.
- Treating a passing test suite as coverage. Guardrails fail on the cases nobody imagined, which is precisely why they're the cases that matter.
At a glance
FieldAI Agents
Core ideabounds around a system that has none
Strongest formreduced permissions
Weakest formasking nicely
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Guardrails vs. alignment — external constraints on a system vs. the system pursuing intended goals in the first place.