14. Red teaming, evaluation, and AI for defense

You cannot prove an AI system secure, so you measure it. Red teaming finds failures nobody imagined; evaluation tells you how often they happen and whether fixes hold. Both only count if they model an adaptive attacker, and both need statistics, because AI systems are not deterministic.

Three activities, often confused

• Penetration testing - an authorized attempt to breach a system’s security, usually focused on infrastructure and application bugs.
• AI red teaming - adversarial probing of an AI system’s behaviour and integrations: injection, jailbreaks, data leakage, harmful outputs, agent misuse. It spans security and safety.
• Evaluations (evals) - repeatable, measured test suites run on every change. They are regression tests for behaviour.

They feed each other: every red-team finding should become an eval case.

A methodology

1. Scope and rules of engagement. Which system and version, which techniques are allowed, how data is handled - and written authorization (Part 17).
2. Threat model. Who attacks, with what goal and access (Parts 1 and 8). Pick categories from OWASP and MITRE ATLAS so coverage is systematic.
3. Test cases. Start with human creativity. Then generate and mutate automatically: paraphrases, encodings, other languages, multi-turn conversations, images and audio. Include indirect injection through documents, web pages and tool outputs. Test the whole system, not the bare model.
4. Execution. Run each case many times; outputs vary from run to run.
5. Measurement. Attack success rate per category, with uncertainty. Also measure what defenses cost: how often they block legitimate users, and how much latency they add.
6. Reporting. Reproducible cases, severity judged by impact (what the attacker achieves, not how clever the prompt was), root cause, and fix.
7. Regression. Add every finding to the eval suite and re-run it whenever the model, prompt, tools or retrieval change.

Measuring honestly

Zero failures is not zero risk. If 0 of n trials succeed, the 95% upper bound on the true success rate is roughly 3/n (the “rule of three”). Zero successes in 50 trials still allows a rate of about 6%.

Attackers get many tries. If one attempt succeeds with probability p, then k independent attempts succeed at least once with probability:

With p = 1% and k = 100 attempts, that is about 63%. A defense that “blocks 99% of attacks” may still fail against one patient attacker. This is why system-level limits (Parts 10–11) matter more than filter accuracy.

Adaptive beats static. A benchmark of known jailbreaks overstates robustness, just as in Part 9. In 2025, researchers from several AI labs used adaptive attacks - optimization plus human red-teamers who studied each defense - to bypass a dozen published jailbreak and prompt-injection defenses, most of them with high success rates. Ask of any defense: was it tested by people trying to break it specifically?

Benchmarks such as AgentDojo (prompt injection in agent tasks) and HarmBench (jailbreaks) are useful starting points. Remember that models and defenses can be tuned to a benchmark without becoming more secure.

Tools worth knowing

Tool	Purpose
garak	Scans LLMs for known vulnerability classes (injection, leakage, jailbreaks)
PyRIT	Automates red-teaming campaigns against generative AI systems
promptfoo	Configurable evals and red-team test suites for LLM apps, CI-friendly
Inspect	Evaluation framework from the UK AI Security Institute
Adversarial Robustness Toolbox (ART)	Classical adversarial ML: evasion, poisoning, extraction, inference attacks and defenses

Guardrails and monitoring in production

• Classifiers on inputs and outputs for injection attempts and policy violations, plus a deterministic policy layer on tool calls.
• Full traces - prompt, retrieved sources, tool calls and outputs - logged with privacy care, so incidents can be reconstructed.
• Canary tokens - unique strings planted in system prompts or sensitive documents. If one ever appears in an output or anywhere outside, you have detected a leak.
• Usage and spend monitoring to catch extraction, abuse and runaway agents.

AI for defense - and offense

Defenders now use LLMs to triage alerts, summarize incidents, review code, write detection rules, analyze malware, and generate fuzzing harnesses. AI-assisted vulnerability discovery has advanced quickly. From 2024, AI agents began finding real vulnerabilities in widely used open-source software, and DARPA’s AI Cyber Challenge (2023–2025) had autonomous systems find and patch bugs.

Attackers use the same technology for phishing at scale, reconnaissance and malware variation. The capability is dual-use, so the decisive factor becomes speed: how fast defenders find and patch compared with how fast attackers find and exploit. Frontier AI developers evaluate their models’ cyber capabilities before release and publish reports on misuse they disrupt.

Questions to ask

• What threat model does this red-team exercise cover, and what does it leave out?
• Was the whole system tested - retrieval, tools, memory - or only the bare model?
• How many trials per case, what success rate, and with what uncertainty?
• Did the testers know the defenses and adapt to them?
• Do findings become regression tests that run on every change?
• Could we reconstruct an incident from what we log in production?
• What false-positive rate and latency do our guardrails add?