Agent Governance
The controls around a system that acts rather than answers — a different problem from model safety, and the field is early.
Reviewed July 14, 2026Stable
When not to use it
- (Controls that give false comfort.)*
- A system prompt as a control. It's a preference in the same channel an attacker writes to.
- The agent's own audit log. It's writing a story. Log from infrastructure.
- Approval gates at high volume. The fortieth approval today is a rubber stamp, and it launders the decision.
- Borrowed user credentials. Your permission model just became "whatever that user could do."
Reach for something else instead
- Sandboxing — the only control that doesn't depend on the model behaving.
- Scoped credentials — solved in security, ignored by agents.
- Read-only agents — if it can't write, most of this disappears.
- Not using an agent — a deterministic pipeline has none of these properties.
Read more on the blog
- Agent permissions: the question nobody asks until afterwardsEighty percent of organisations running agents say those agents have taken unintended actions. One in five has had a security incident from one. Both wrong answers to "whose credentials" are still the common ones.
- Why AI agents fail: the seven failure modesGartner predicts over 40% of agentic AI projects will be canceled by 2027. The failures follow patterns, seven of them. The taxonomy: what breaks, why, which real incident proved it, and which control would have prevented it.
- Human in the loop is weaker than it soundsWhen the AI was wrong, experienced radiologists went from 82% accurate to 45.5%. A review step is only a control if the reviewer sometimes disagrees, and almost nobody measures how often they do.
- Most agent projects will be cancelled. This is why.Gartner puts the cancellation rate above 40% by the end of 2027. None of the three stated causes is a model problem, and the most common one is that the use case never needed an agent.
Further reading
- Shavit et al. (2023), Practices for Governing Agentic AI Systems — the first serious attempt at the discipline.
- Chan et al. (2023), Harms from Increasingly Agentic Algorithmic Systems — why acting is a different problem from answering.
- Willison (2023–), Prompt injection series — the structural argument for why instruction-based controls cannot work.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Instructing the agent instead of constraining it. Prompt injection is structural; instructions aren't controls.
- Giving it a user's credentials. Now the agent's authority is that user's, and so is an attacker's.
- Building approval gates that fire constantly. Volume destroys the control.
- Treating this as an AI problem. It's access control with a non-deterministic principal.
At a glance
FieldAI Agents
Why it's distinctan agent's output is an action, and actions don't wait for review
The default flawinstructions and data share a channel, so anything it reads can instruct it
What workssandboxing and scoped credentials
What doesn'ttelling it not to
The framingit's access control, not AI safety
DifficultyIntermediate
Often compared with
Agent governance vs. model alignment — one asks what the system is allowed to do and enforces it outside the model; the other asks what the model wants and hopes. Only the first survives prompt injection.
Where this sits
A destination. 40 concepts lead here, and nothing in the corpus depends on it.
17Levelsteps in
40Needs firstconcepts
0Opens upnothing further
1Areastays here
Computed from the prerequisite graph, not assigned. How this works