8. The bridge: why AI systems are different

AI systems inherit every classical security problem from Parts 1–7, then add a few genuinely new ones. The deepest change: behaviour is learned from data rather than written in code, so whoever influences the data influences the program - and there is no precise specification to check that behaviour against.

What actually changes

1. Data is code. In ordinary software you can read the code that defines behaviour. In ML, the training data defines it, and the result is buried in billions of weights. Influence over data is influence over behaviour.
2. No specification. You cannot write down the correct output for every possible input; the model is right only statistically. You cannot prove “it never does X”. Defenses become probabilistic and are measured by attack success rate, not proofs.
3. Enormous input spaces. Images, audio and text offer an attacker a vast, continuous space to search for failures. For white-box attackers, gradients point straight at them.
4. One channel for instructions and data. For LLMs, any text the model reads may steer it. This is Part 2’s injection problem with no structural fix.
5. Opaque internals. You cannot inspect weights the way you review code, so a backdoor can hide in plain sight.
6. Memorization. Models can store and later reveal pieces of their training data.
7. Non-determinism. The same input can yield different outputs, so one passing test proves little. Testing must be statistical.
8. Capability raises stakes. The more an agent can do, the more harm it does when misdirected.

The ML lifecycle as attack surface

Each stage of the lifecycle has its own attackers, and the diagram below maps them. Two things stand out. Training-time attacks (poisoning, backdoors) are planted early and triggered much later, long after anyone could notice. And the integration stage - where models gain retrieval, tools and memory - is where most real-world LLM incidents happen today.

[embedded content: ML lifecycle · 8 stages, attacks at each]

Read it as a loop: what users and the world feed back becomes the next round’s training data, so a deployment-time input can turn into a training-time attack.

A taxonomy to organize everything

NIST’s report on adversarial machine learning (NIST AI 100-2, updated in 2025) organizes attacks along a few dimensions. Learn these and you can place any new paper or incident in seconds.

• Attacker goal - integrity (make it do the wrong thing), availability (make it fail or cost too much), privacy (extract data or the model), and, for generative AI, misuse (make it produce what its operator forbade).
• Attacker knowledge - white-box (weights and architecture), grey-box (partial), black-box (query access only).
• Attacker capability - control over training data, the model itself, queries, source code, or the resources a model reads at runtime (web pages, documents, emails).
• Lifecycle stage - training time or deployment time.

Attack	When	Goal	Typical attacker access
Evasion (adversarial examples)	Deployment	Integrity	Queries; gradients if white-box
Data poisoning	Training	Integrity or availability	Some training data
Backdoor / trojan	Training or supply chain	Integrity (on a secret trigger)	Training data or the model file
Membership inference	Deployment	Privacy	Queries, often with confidence scores
Model inversion / data extraction	Deployment	Privacy	Queries
Model extraction (stealing)	Deployment	Privacy / intellectual property	Many queries
Direct prompt injection / jailbreak	Deployment	Misuse	Chat access
Indirect prompt injection	Deployment	Integrity, privacy, misuse	Content the model will read

Frameworks you will hear about

• MITRE ATLAS - an ATT&CK-style matrix of real tactics and techniques against AI systems, with case studies. Use it as a threat-modeling checklist.
• NIST AI 100-2 - the reference taxonomy and vocabulary for adversarial ML.
• OWASP GenAI Security Project - the Top 10 lists for LLM applications and for agentic applications (Parts 10–11).
• NIST AI RMF, ISO/IEC 42001 and the EU AI Act - governance and regulation (Part 13).
• Germany’s BSI publishes guidance on AI and LLM security risks, useful when working with German organizations.

Security, safety, alignment: three words people mix up

• AI security protects AI systems from adversaries, and protects everything else from adversaries who use or abuse AI.
• AI safety is about AI systems not causing harm, with or without an attacker.
• Alignment is about making a model’s goals and behaviour match what its developers intend.

They overlap. A jailbreak is a security attack that defeats alignment training and produces a safety failure. When reading a paper or job description, check which of the three it actually means.

Questions to ask

• Who can influence the data this model learns from - now and at the next retraining?
• At which lifecycle stages does untrusted input enter?
• What does the attacker know and control: weights, queries, or content the model reads?
• Is a claimed defense measured against an adaptive attacker, and with what success rate?
• Which failures here are security failures, which are safety failures, and who owns each?