Red-teaming
Attacking your own system on purpose, before someone else does it for free.
When not to use it
- As a certificate. "We red-teamed it" describes an activity, not a property of the system.
- Only internally. The people who built it share its blind spots and are motivated not to find things.
- Without a remediation path. Findings that go into a document nobody owns are a record of what you knew and didn't fix.
Reach for something else instead
- Reduced scope — cheaper and more effective than testing whether a dangerous capability can be abused.
- Formal constraints — schema-bound output, permission limits, hard budgets. Testable properties beat adversarial hope.
- Staged rollout with monitoring — real users find things no red team imagined; the point is catching it early rather than at scale.
Sources & further reading
- Ganguli et al. (2022), Red Teaming Language Models to Reduce Harms — a large-scale effort described honestly, including what it missed.
- Perez et al. (2022), Red Teaming Language Models with Language Models — automating the attacker.
- Zou et al. (2023), Universal and Transferable Adversarial Attacks on Aligned Language Models — why manual red-teaming alone is now insufficient.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Counting findings instead of assessing severity, which rewards finding many harmless things.
- Red-teaming the model instead of the system. Incidents come from what it was wired to.
- Treating it as a launch gate rather than an ongoing practice. Capability changes reopen doors.
At a glance
FieldSafety & Ethics
Core ideaattack your own system deliberately
Best done bypeople who didn't build it
Outputa map of reachable harm, not a count
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Red-teaming vs. guardrails — searching for the holes vs. the walls you built.