ML Security: Adversarial Attacks

Models can be fooled on purpose, not just confused by chance

A model's accuracy on natural, randomly sampled inputs says nothing about how it behaves under inputs deliberately engineered to exploit its decision boundary. Adversarial examples are inputs modified with a small, often imperceptible perturbation specifically designed to cause a misclassification — an image of a stop sign with a few pixels altered that a human still clearly sees as a stop sign, but that a vision model confidently labels as something else. These aren't edge cases that occur by chance; they're found deliberately by an attacker (or a researcher) with access to the model's gradients or enough query access to approximate them.

ML security treats this as a distinct threat category from traditional application security, because the vulnerability lives in the model's learned decision boundary itself, not in a code bug that can be patched with a single fix.

Three main attack categories

Evasion attacks happen at inference time: the attacker crafts an input designed to be misclassified by an already-trained, already-deployed model — the classic adversarial example scenario, relevant to spam filters, malware detectors, fraud models, and vision systems alike. Poisoning attacks happen at training time: the attacker injects corrupted or mislabeled examples into the training data (or into a data source the model will be retrained on), so the model itself learns a flawed or backdoored decision boundary — particularly dangerous for systems that retrain continuously on user-submitted or crowd-sourced data. Model extraction attacks target the model itself as the asset: by querying a deployed model's API repeatedly and observing its outputs, an attacker can train a substitute model that closely mimics the original's behavior, effectively stealing the intellectual property (and sometimes using the substitute to craft evasion attacks offline, without burning queries against the real system).

Adversarial attack categories

Defense strategies

Adversarial training — explicitly generating adversarial examples during training and including them in the training set — is the most directly targeted defense against evasion attacks, forcing the model to learn a decision boundary that's robust to small perturbations, though it typically costs some accuracy on clean, non-adversarial inputs. Input validation and anomaly detection can catch inputs that are statistically unusual compared to normal traffic, flagging potential adversarial inputs for review even without stopping them outright. Data provenance and validation pipelines defend against poisoning by auditing and filtering training data sources, especially user-submitted or third-party data, before it reaches a retraining job. Rate limiting and query monitoring on model-serving APIs make model extraction and the systematic probing that evasion attack development often requires more expensive and detectable, without eliminating the risk entirely. No single defense is complete, and robustness against one attack type doesn't transfer to robustness against the others.

What's next

Model security protects the model from attackers trying to manipulate it; a related but distinct concern is protecting the data it was trained on from ever being exposed in the first place. Continue to Privacy-Preserving Machine Learning to see how differential privacy, secure aggregation, and encryption let models train without exposing raw data.