Adversarial Attack
A deliberately crafted input that fools an AI model — a few pixels or words, invisible or innocuous to humans, that flip the model's answer completely.
Reviewed July 16, 2026Stable
When not to use it
- (Adversarial attacks are a threat to defend against, not a tool to deploy — except in red-teaming, where crafting them is exactly how you test robustness.)
Reach for something else instead
- Adversarial training — the leading defence, training on attacked examples.
- Certified/provable robustness when you need guarantees within a bounded perturbation (at an accuracy cost).
- Input detection and preprocessing as partial, defeatable mitigations.
A tiny added perturbation flips the model's answer.
Add a faint, structured noise pattern — imperceptible to a human — to an image, and a classifier that was certain it saw a panda now confidently says gibbon. The input barely changed for us; it changed decisively for the model, because it responds to statistical patterns we can't see. That's the core of an adversarial attack.
Sources & further reading
- Szegedy et al. (2013), Intriguing Properties of Neural Networks — the discovery of adversarial examples.
- Goodfellow et al. (2014), Explaining and Harnessing Adversarial Examples — the fast gradient sign method and the linearity explanation.
- Madry et al. (2017), Towards Deep Learning Models Resistant to Adversarial Attacks — adversarial training as a principled defence.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Evaluating a security-critical model only on honest inputs, ignoring hostile ones.
- Trusting a defence that wasn't tested against adaptive attacks — many broke when actually attacked.
- Assuming a black-box model is safe — adversarial examples transfer across models.
At a glance
FieldSafety & Ethics
What it iscrafted input that fools a model
Classic methodgradient-based perturbation
Modern formprompt injection, jailbreaks
DifficultyIntermediate