9. Adversarial machine learning

Models can be fooled at inference time (evasion), corrupted during training (poisoning and backdoors), and made to leak their training data or themselves (privacy and extraction attacks). A decade of research teaches one hard lesson: most proposed defenses fail once an attacker adapts to them, and the few that hold come with real costs.

Evasion: adversarial examples

In 2013–2014, researchers showed that tiny, often invisible changes to an image can make a classifier output a wrong label with high confidence. The same holds for audio, text, malware detectors, fraud models and spam filters.

Why they exist - the intuition. Take a linear score w·x over n input features. Nudge every feature by a tiny amount ε in the direction of its weight’s sign. Each nudge is invisible, but the score shifts by ε times the sum of all |wᵢ|, which grows with the number of features. In high-dimensional inputs, thousands of imperceptible changes add up to a large change in output. Later work showed models also rely on real but brittle “non-robust features” that predict labels well yet mean nothing to humans.

The formal problem. The attacker looks for a small perturbation δ, within a budget ε under some norm, that maximizes the model’s loss:

The threat model is the choice of norm and budget - and it matters enormously. Robustness against small pixel changes says nothing about robustness against a printed sticker.

How attacks work, conceptually:

• White-box gradient attacks. FGSM takes one step along the sign of the loss gradient. PGD takes many small steps, projecting back into the allowed budget after each. PGD is the standard strong baseline.
• Transfer attacks. Adversarial examples built on a model you control often fool a different model trained for the same task. Black-box access is therefore less protective than it seems.
• Query-based attacks. The attacker estimates gradients purely from the model’s outputs.
• Physical attacks. Patches, stickers and patterned glasses that survive printing, angles and lighting.
• Constrained domains. For malware or fraud, the modified input must stay valid (the malware must still run), so attacks search a constrained feature space.

Defenses, and what survived:

• Adversarial training - train on adversarial examples, turning training into a min–max game. It is the most reliable empirical defense, but it is expensive, lowers clean accuracy, and protects only against the threat model it was trained for.

• Certified defenses (such as randomized smoothing) prove that no perturbation within a radius can change the prediction. The guarantees are real but cover small radii.
• Detectors, input transformations, and gradient obfuscation - mostly broken. In 2018, researchers circumvented seven of nine defenses published at a top conference by attacking around their obfuscated gradients. Hiding gradients only defeats lazy attackers.

The transferable lesson, which applies directly to LLM guardrails: a defense is only as good as its evaluation against an adaptive attacker who knows the defense. Be suspicious of any defense evaluated only on yesterday’s attacks.

Poisoning and backdoors

Poisoning attackers control part of the training data.

• Availability poisoning degrades the model overall.
• Targeted poisoning makes specific inputs fail.
• Backdoors (trojans) make the model behave normally - until a secret trigger appears (a pixel pattern, a rare phrase), and then produce the attacker’s chosen output. Clean test accuracy is unaffected, so normal evaluation never finds it.
• Clean-label poisoning uses samples that look correctly labeled to a human reviewer.

Four research results show why this matters at scale:

• Poisoning web-scale datasets is cheap. A 2023 study showed that many image datasets distributed as URL lists pointed to expired domains. Buying those domains would have let an attacker serve different images. The authors estimated that about $60 would have poisoned 0.01% of LAION-400M.
• A near-constant number of documents can suffice. A 2025 study by Anthropic, the UK AI Security Institute and the Alan Turing Institute found that around 250 poisoned documents planted a simple backdoor in LLMs from 600M to 13B parameters. The count did not grow with model size, so bigger datasets do not dilute the attack.
• Backdoors can survive safety training. Anthropic’s “Sleeper Agents” work (2024) trained models with hidden triggered behaviours that persisted through standard safety fine-tuning.
• Fine-tuning can strip safety. Fine-tuning an aligned model on a handful of harmful - or even benign - examples can weaken its safety behaviour. Fine-tuning access is an attack surface.

Defenses: control data provenance and who can write to datasets; verify hashes of downloaded data; deduplicate and look for outliers; limit how much any single source can influence training; restrict access to the training pipeline; and test released models for unexpected triggered behaviour. Backdoor detection remains an open research problem, so prevention matters more than detection.

Privacy attacks

• Membership inference asks: was this record in the training set? Models tend to be more confident on data they trained on, and attackers exploit that gap. Membership alone can be sensitive - being in a hospital’s training set reveals you were a patient. The strongest modern attacks are measured at very low false-positive rates, because confident identification of a few people is what matters. Membership inference is also the standard tool for auditing privacy leakage.
• Model inversion and attribute inference reconstruct sensitive features or representative inputs, such as a recognizable face from a face-recognition model.
• Training-data extraction. Researchers extracted verbatim memorized text - names, emails, phone numbers - from GPT-2 in 2021. In 2023, a simple prompt that made a production chatbot repeat one word endlessly caused it to diverge and emit memorized training data. Memorization grows with model size and with how often a sequence is duplicated in the data.
• Gradient leakage. In federated learning, the gradients clients share can be used to reconstruct their training inputs.

Model extraction

An attacker queries a model and uses the answers to train a functional copy. Motives: steal intellectual property, avoid paying, or obtain a white-box copy for crafting transfer attacks against the original. Research in 2024 even recovered part of a production LLM’s final layer purely through its API.

Defenses: rate limits and query monitoring; returning less (labels instead of full probabilities, no log-probabilities unless needed); watermarking outputs; and contractual and legal controls.

Differential privacy: privacy you can prove

Differential privacy (DP) is a property of an algorithm: its output barely changes whether or not any single individual’s data is included. So the output cannot reveal much about any one person. Formally, for any two datasets D and D′ differing in one individual:

• ε (epsilon) is the privacy budget: smaller means stronger privacy. Budgets add up across repeated releases (composition).
• DP-SGD makes training private: clip each example’s gradient to bound its influence, add calibrated Gaussian noise, and track the total budget with a privacy accountant. Opacus implements this for PyTorch.
• Costs: lower accuracy and more compute. It works best with large datasets, often by pretraining on public data and fine-tuning privately.
• Limits: DP protects individuals, not facts shared by many people. The guarantee depends on the unit (one record, or one user with many records). And the ε values used in practice vary widely, so always ask for the number.

Two related notes. Federated learning is not private by itself; it needs secure aggregation and DP on top. Machine unlearning tries to remove a person’s data from a trained model, for example after a GDPR erasure request. Exact unlearning means retraining; approximate methods are cheaper but hard to verify.

Questions to ask

• What is the threat model of this robustness claim: which perturbations, what budget, what attacker knowledge?
• Was the defense tested with adaptive attacks by people actively trying to break it?
• Who could have contributed training data, and could they have planted triggers?
• What does the API reveal: labels, confidence scores, log-probabilities, embeddings?
• Could the model reveal whether a specific person’s data was used, or repeat it verbatim?
• If someone claims differential privacy: what ε, over what unit, and at what accuracy cost?