Cybersecurity for AI Systems from First Principles

Security is one idea applied again and again: a system is secure only relative to a specific adversary, with specific capabilities, trying to break a specific property. Every part below rebuilds that idea for one layer of the stack, ending with AI systems.

How to use this guide

The guide assumes you can already program, build web apps and train models. It does not assume any security background. The goal is not to memorize attack lists. It is to reason from the foundations well enough to ask sharp questions, judge answers, and talk fluently with security engineers and AI researchers.

Why classical security first

Most real AI incidents are classical failures in new clothing: leaked API keys, over-permissioned service accounts, unsafe deserialization of model files, injection. A small set of problems is genuinely new (adversarial examples, poisoning, prompt injection). You can only see what is new once you know what is old. So Parts 1–7 build the foundation, Parts 8–14 cover AI, and Parts 15–17 turn it into practice.

Three threads that run through everything

1. Code versus data. Most injection attacks happen when a system treats data as instructions. SQL injection, cross-site scripting, unsafe deserialization and prompt injection are the same mistake at different layers. LLMs are the extreme case: in natural language, there is no hard line between data and instructions at all.
2. Authority and trust. Who is allowed to cause what to happen? Most breaches are someone (or something) acting with authority it should not have. The “confused deputy” from Part 5 becomes the central problem of AI agents in Part 11.
3. Assumptions. Every attack is a violated assumption: “users only send numbers here”, “the training data is clean”, “the model only follows my system prompt”. Security work is largely the craft of finding assumptions and asking what happens when they are false.

How each part is built

Each part explains the core idea, the mechanism behind it, how it fails, and where it shows up in ML and LLM systems. Each ends with Questions to ask - the questions a practitioner would ask in a design review. Those question lists are the most reusable part of the guide.

The learning loop

1. Read one part slowly.
2. Close the guide and explain its core idea out loud in a few sentences. If you can’t, re-read the mechanism.
3. Do one hands-on exercise from Part 17 that matches the part.
4. Apply the part’s questions to something you have built yourself, and write down the answers.

Step 4 is where fluency comes from. Reading about trust boundaries teaches you vocabulary. Finding one in your own code teaches you the concept.

Scope

This guide is defensive and conceptual. It explains how attacks work at the level needed to design defenses and evaluate claims. It does not contain working exploit code. Only ever test systems you own or have written permission to test; Part 17 explains why this matters legally in Germany.