Agent Governance
The controls around a system that acts rather than answers — a different problem from model safety, and the field is early.
When not to use it
- (Controls that give false comfort.)*
- A system prompt as a control. It's a preference in the same channel an attacker writes to.
- The agent's own audit log. It's writing a story. Log from infrastructure.
- Approval gates at high volume. The fortieth approval today is a rubber stamp, and it launders the decision.
- Borrowed user credentials. Your permission model just became "whatever that user could do."
Reach for something else instead
- Sandboxing — the only control that doesn't depend on the model behaving.
- Scoped credentials — solved in security, ignored by agents.
- Read-only agents — if it can't write, most of this disappears.
- Not using an agent — a deterministic pipeline has none of these properties.
Sources & further reading
- Shavit et al. (2023), Practices for Governing Agentic AI Systems — the first serious attempt at the discipline.
- Chan et al. (2023), Harms from Increasingly Agentic Algorithmic Systems — why acting is a different problem from answering.
- Willison (2023–), Prompt injection series — the structural argument for why instruction-based controls cannot work.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Instructing the agent instead of constraining it. Prompt injection is structural; instructions aren't controls.
- Giving it a user's credentials. Now the agent's authority is that user's, and so is an attacker's.
- Building approval gates that fire constantly. Volume destroys the control.
- Treating this as an AI problem. It's access control with a non-deterministic principal.
At a glance
FieldAI Agents
Why it's distinctan agent's output is an action, and actions don't wait for review
The default flawinstructions and data share a channel, so anything it reads can instruct it
What workssandboxing and scoped credentials
What doesn'ttelling it not to
The framingit's access control, not AI safety
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Agent governance vs. model alignment — one asks what the system is allowed to do and enforces it outside the model; the other asks what the model wants and hopes. Only the first survives prompt injection.