17. Learning path, practice, reading list

Reading builds vocabulary; building and breaking your own systems builds understanding. This part gives a 12-week path at roughly 6–8 hours per week, the hands-on resources behind it, a reading list, and the legal ground rules you must follow while practising.

Ground rules first: legal and ethical

• Only test what you own or have written permission to test. German criminal law covers unauthorized access to protected data, interception, preparing such acts, data manipulation and computer sabotage (§§ 202a–202c, 303a–303b StGB). Good intentions are not a defense to rely on.
• Use legal playgrounds: your own machines and cloud accounts, deliberately vulnerable labs, and bug bounty programs strictly within their published scope.
• AI services have terms of use. Probing a production AI product beyond normal use can breach them. Many providers run security and safety disclosure programs; use those channels.
• Disclose responsibly. If you find a real vulnerability, report it privately to the vendor and give them time to fix it before discussing it publicly.

A 12-week path

Weeks	Read	Do	Produce
1	How to use; Part 1	Draw a data-flow diagram of one of your own projects and walk it through STRIDE	One-page threat model
2	Part 2	OverTheWire Bandit (Linux and permissions); disassemble a harmless pickle with Python’s pickletools to see why loading it can call functions	Notes on privilege and isolation
3	Part 3	Inspect your own traffic with Wireshark; read a site’s TLS certificate chain	A diagram of what an observer sees
4	Part 4	Cryptopals set 1; audit your own code for weak randomness, hashing and stored secrets	Fixes in your repositories
5	Part 5	PortSwigger Web Security Academy: access control, authentication and OAuth labs	Lab write-ups
6	Parts 6–7	PortSwigger: SQL injection, XSS and SSRF labs; run pip-audit, Semgrep and a secret scanner on your repos	Triaged findings, fixed
7	Parts 8–9	Train a small image classifier; attack it with FGSM and PGD using ART; then adversarially train it	Notebook: accuracy vs robustness
8	Part 9	Train with and without DP-SGD (Opacus); run a membership-inference attack on both	Notebook: privacy vs accuracy
9	Part 10	Play Lakera’s Gandalf; build a small RAG app with per-user permissions; attack it with poisoned documents; fix it	Before/after attack success rates
10	Part 11	Run AgentDojo; build a tiny agent using the plan-then-execute or dual-LLM pattern	Short design note on what it blocks
11	Parts 12–13	Make a project load safetensors only, pin model hashes, and generate an AIBOM; read AI Act Article 15	A supply-chain checklist, applied
12	Parts 14–16	Red-team your RAG app with garak or promptfoo; repeat Part 15 for your own system	A red-team report for your portfolio

Hands-on resources

Resource	What it teaches	Level
OverTheWire: Bandit	Linux, shells, permissions, basic tooling	Beginner
PortSwigger Web Security Academy	Every major web vulnerability, with free interactive labs	Beginner to advanced
OWASP Juice Shop	A deliberately vulnerable web app to run locally	Beginner to intermediate
picoCTF	Capture-the-flag challenges across many areas	Beginner
TryHackMe / Hack The Box	Guided rooms and realistic machines	Beginner to advanced
Cryptopals	Breaking real crypto mistakes by implementing attacks	Intermediate
Lakera Gandalf	Prompt-injection intuition, as a game	Beginner
AgentDojo	Prompt injection against tool-using agents, measured	Intermediate
Adversarial Robustness Toolbox (ART)	Classical adversarial ML attacks and defenses in Python	Intermediate
Opacus	Differentially private training in PyTorch	Intermediate

Reading list

Foundations

• Ross Anderson, Security Engineering (3rd edition, 2020) - the best single book on the field; many chapters are free online.
• Saltzer and Schroeder, “The Protection of Information in Computer Systems” (1975) - the design principles from Part 1.
• Norm Hardy, “The Confused Deputy” (1988) - two pages that explain agent security decades early.
• Adam Shostack, Threat Modeling: Designing for Security (2014).
• Jean-Philippe Aumasson, Serious Cryptography (2nd edition, 2024) - practical crypto without heavy math.

Adversarial ML and privacy

• NIST AI 100-2, Adversarial Machine Learning: A Taxonomy and Terminology (2025 edition).
• Goodfellow, Shlens and Szegedy, “Explaining and Harnessing Adversarial Examples” (2014).
• Madry et al., “Towards Deep Learning Models Resistant to Adversarial Attacks” (2018).
• Athalye, Carlini and Wagner, “Obfuscated Gradients Give a False Sense of Security” (2018).
• Shokri et al., “Membership Inference Attacks Against Machine Learning Models” (2017).
• Carlini et al., “Extracting Training Data from Large Language Models” (2021).
• Carlini et al., “Poisoning Web-Scale Training Datasets is Practical” (2023).
• Dwork and Roth, The Algorithmic Foundations of Differential Privacy (2014, free online).
• Abadi et al., “Deep Learning with Differential Privacy” (2016).

LLMs and agents

• Greshake et al., “Not What You’ve Signed Up For” (2023) - indirect prompt injection.
• Zou et al., “Universal and Transferable Adversarial Attacks on Aligned Language Models” (2023).
• Debenedetti et al., “AgentDojo” (2024).
• Beurer-Kellner et al., “Design Patterns for Securing LLM Agents against Prompt Injections” (2025).
• Debenedetti et al., “Defeating Prompt Injections by Design” (2025) - the CaMeL system.
• Simon Willison’s blog - the most consistent running commentary on prompt injection and agent security.

Living references: the OWASP GenAI Security Project, MITRE ATLAS, and publications from Germany’s BSI.

On certifications

General certifications such as CompTIA Security+ (foundations) or OSCP (offensive practice) are recognized in security hiring. AI security has no dominant certification yet. A portfolio of threat models, red-team reports and build-and-break notebooks - like the outputs in the table above - demonstrates fluency more directly.