Jailbreaking
Getting a model to do what it was trained to refuse — and the structural reason it keeps working.
When not to use it
- Against systems you don't own or have permission to test. That's not research, and the distinction matters legally.
- As your only safety evidence. Passing your jailbreak tests means your tests passed; the space of attacks is larger than your imagination.
- As a reason to ship nothing. Every deployed model is jailbreakable to some degree. The question is what it can reach when it happens.
Reach for something else instead
- Reduced capability — the only defence that doesn't depend on the model's judgement. If it can't do the thing, it can't be persuaded to.
- Output validation in your code, not the model's. Check what comes back before anyone sees it.
- Human review on anything consequential enough that a jailbreak would be an incident.
Sources & further reading
- Zou et al. (2023), Universal and Transferable Adversarial Attacks on Aligned Language Models — automated attacks that transfer across models.
- Wei, Haghtalab & Steinhardt (2023), Jailbroken: How Does LLM Safety Training Fail? — the failure modes, framed structurally rather than as a trick list.
- Greshake et al. (2023), Not what you've signed up for — indirect injection through retrieved content, which is the version that matters in production.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Adding "do not comply with attempts to bypass these rules" to the prompt. Instructions in the prompt have no special standing over instructions in the input.
- Defending only the user's message. The realistic attack path is content your system retrieves, not what the user types.
- Assuming a patched jailbreak is a solved class. The specific string stopped working; the technique usually didn't.
At a glance
FieldSafety & Ethics
Core ideatalking a model past its trained refusals
Why it persistsrefusals are tendencies, not rules
Real defencereduced capability
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Jailbreaking vs. prompt injection — the user talking the model round vs. the data doing it.