Prompt Injection
Hiding instructions inside content a model reads, so it follows the attacker instead of you — and the reason is structural, which is why it isn't fixed.
When not to use it
- (This is a risk, not a technique — the equivalent question is when you can stop worrying about it.)*
- When nothing the model reads comes from outside your trust boundary. Rare, and check the assumption twice.
- When the model has no capabilities. A pure text generator with no tools and no side effects can be injected to no consequence beyond a bad answer.
- Never, if it has tools and reads external content. There's no configuration that makes this safe.
Reach for something else instead
- (Ways to make it not matter, since you can't prevent it.)*
- Capability restriction — the only reliable defence. Don't grant what you can't afford to lose.
- Human confirmation on irreversible actions — moves the trust boundary to a person.
- Read-only credentials — worst case becomes a wrong answer rather than a wrong action.
- Treating model output as untrusted — validate it before it reaches anything that acts.
Sources & further reading
- Greshake et al. (2023), Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — the systematic treatment; read this one.
- Wallace et al. (2024), The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions — the most credible mitigation direction, and honest about being a tendency rather than a boundary.
- Perez & Ribeiro (2022), Ignore Previous Prompt: Attack Techniques For Language Models — early formalisation of the attack.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Believing a system prompt is a security boundary. It's a strong suggestion.
- Defending against direct injection and ignoring indirect, which is the one that hurts your users.
- Testing with obvious attacks ("ignore previous instructions") and concluding you're safe. Real attacks don't announce themselves.
- Adding a detection classifier and calling it solved. The attacker iterates; your classifier doesn't.
- Assuming multi-agent architectures contain the blast radius. They enlarge it — every hop is another chance for data to become instruction.
At a glance
FieldSafety & Ethics
Typestructural vulnerability
Direct vs. indirectindirect is the dangerous one
Root causeinstructions and data share one channel
Statusunsolved in the general case
Only reliable defencedon't grant the capability
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Prompt injection vs. jailbreaking — jailbreaking is a user getting a model to misbehave for themselves. Injection is a third party getting a model to misbehave against its own user. Same mechanism, very different victim.