Guardrails

The full account

The question guardrails answer

A language model will attempt anything. It has no notion of what's off-limits, no knowledge of your policies, and — this is the load-bearing fact — no reliable way to tell whether an instruction came from you or from a document it happened to read. Everything called a "guardrail" exists because of that last sentence. The model cannot police itself, so the policing has to happen around it.

The engineering breaks into three layers, and it's worth being precise about them because vendors blur them constantly. Input guardrails decide what reaches the model: filtering user messages, screening retrieved documents, stripping or flagging suspicious instructions. Output guardrails decide what leaves: checking responses for policy violations, leaked secrets, unsupported claims, or formats that would break downstream code. Action guardrails decide what the system may do: which tools can be called, with what arguments, spending what budget, touching which data. The first two are about words. The third is about consequences — and it's the one that matters most the moment a model stops answering questions and starts doing things.

Why the obvious approach keeps failing

The intuitive design is a filter: a classifier, a blocklist, or another model that inspects text and rejects the bad kind. Filters are genuinely useful and every serious deployment runs them. But two structural facts cap what they can achieve.

First, the attack surface is language itself. Perez and Ribeiro showed in 2022 that simple instructions like "ignore previous directions" could hijack deployed models — no exploit code, just sentences. Blocklists respond by enumerating bad strings, and language responds by having infinitely many ways to say anything. The filter is always finite; the paraphrases are not.

Second, and more fundamentally: the model cannot distinguish data from instructions. This is the finding of Greshake and colleagues' 2023 paper on indirect prompt injection, and it's the single most important result in this area. Instructions don't have to come from the user. They can arrive inside the content the model processes — a web page it browses, a document it summarises, a ticket it reads, an email it triages. The paper demonstrated real compromises of deployed systems this way, including exfiltrating user data through a browsing agent, and laid out a taxonomy — retrieved content, tool outputs, poisoned data — that has held up as agents multiplied. To a transformer, everything in the context window is just tokens; there is no privileged channel marked "these tokens are trusted." Filters inspect content, but injection is content. The defence and the attack occupy the same channel.

That's why the honest engineering posture, increasingly standard in agent security, is: assume injection succeeds. Not "might" — assume it did. Then ask what the blast radius is. That question can't be answered by a text filter, because it isn't about text. It's about permissions.

From guardrails to governance

Follow that logic and you leave guardrails territory. If an injected agent's damage depends on what it could do, then the real controls are the ones that bound its capabilities: which tools it holds, scoped to which resources, spending against which budget, with which actions requiring a human sign-off, all logged where the log can't be quietly edited. Least privilege, approval gates, sandboxing, audit — the vocabulary is access control, not content moderation.

This is the boundary between guardrails and agent governance, and the distinction is not pedantic. Guardrails are per-request: they inspect this input, this output, this action, now. Governance is systemic: identity, permission, budget, and accountability for a principal that happens to be non-deterministic. A team that ships filters and calls the job done has secured the words and left the consequences open. The incidents that make the news are almost never "the model said something rude past the filter." They're "the agent had standing permission to do the thing the attacker asked it to do."

There's also a quieter reason the governance layer matters: models change underneath you. A guardrail tuned to one model's failure modes silently mistunes when the provider ships an update. Permissions don't have that problem. A budget cap holds regardless of which model spends against it; an approval gate doesn't care how eloquent the request was. Deterministic controls age better than probabilistic ones — which is an argument for putting the load-bearing safety on the deterministic side.

What model-side safety buys, and what it can't

None of this makes model-side work useless. Techniques like Constitutional AI — training a model against an explicit set of principles, using model-generated critiques instead of only human labels — genuinely shift baseline behaviour, and a model that mostly refuses harmful requests is a better starting point than one that doesn't. But training shapes dispositions, and dispositions are exactly what injection attacks route around: the compromised agent isn't misbehaving by its own lights, it's obediently following instructions it wrongly attributed to the user. Alignment reduces how often you need the guardrails. It does not replace them, and neither replaces governance.

What to actually build

The practical stack, in order of load-bearing-ness: scope the tools first — an agent that can only read can only leak, and an agent that can't spend can't be spent. Gate the irreversible — payments, deletions, external sends go through a human or a hard rule. Treat all retrieved content as untrusted input, because Greshake showed it is. Then add the input and output filters, which catch the casual majority of problems cheaply. And log everything, because the guardrail you'll want most after an incident is the ability to know exactly what happened.

The one-sentence version: filters police what the model says; governance bounds what it can do; and because the model can't tell data from instructions, the second is the one your incident report will be about.