Cybersecurity for AI Systems from First PrinciplesPost 7 of 19 · 19 of 19 written
- How to use this guide
- 01What security actually is
- 02How computers fail
- 03Networks
- 04Cryptography
- 05Identity and access
- 06Web and application security
- 07Cloud, infrastructure, supply chain, detection
- 08The bridge: why AI systems are different
- 09Adversarial machine learning
- 10LLM and generative AI security
- 11Agentic systems
- 12The ML supply chain and MLOps
- 13Privacy, governance and regulation
- 14Red teaming, evaluation, and AI for defense
- 15Worked example: threat-modeling an AI agent
- 16The question toolkit
- 17Learning path, practice, reading list
- 18Glossary
- 19Sources
7. Cloud, infrastructure, supply chain, detection
In the cloud, identity is the perimeter: whoever holds the right credentials controls the infrastructure. Most cloud breaches come from over-broad permissions, public resources and stolen keys, not exotic exploits. And since no prevention is perfect, the ability to detect and respond is part of the design.
Shared responsibility
The provider secures the physical data centre, the hypervisor and the internals of managed services. You secure your configuration, identities, data and code. Moving from virtual machines to managed platforms to SaaS shifts more work to the provider, but identity and data configuration always remain yours.
Cloud identity and access (IAM)
Cloud IAM has principals (users, roles, service accounts), policies (which actions on which resources), and conditions (when, from where). Typical failures:
- Wildcard permissions (“any action on any resource”) granted to make something work.
- Long-lived access keys on laptops and in CI.
- Storage buckets or disk snapshots made public by accident.
- Privilege-escalation paths: a role that can create or modify other roles can grant itself anything.
- Over-broad trust between accounts or projects.
Good practice: one identity per workload, short-lived role credentials, separate accounts or projects for development and production, and regular removal of unused permissions.
AI link. Training jobs often receive read-write access to an entire data lake “to make it work”. An inference service should be able to read its own model, and nothing else.
Containers and Kubernetes, the essentials
- Images: minimal base images, scanned for known vulnerabilities, pinned by digest, no secrets baked in.
- Runtime: run as non-root, read-only file system, drop Linux capabilities, never “privileged” mode, set resource limits.
- Kubernetes: use RBAC; add network policies (the default allows all pod-to-pod traffic); note that Kubernetes secrets are only base64-encoded unless encryption is configured; use admission control to enforce rules.
- GPUs: GPU drivers are privileged code, and sharing GPUs across tenants raises its own isolation questions.
With infrastructure as code (Terraform and similar), security review happens on the code, scanners can flag risky configuration before deployment, and drift detection catches manual changes.
The software supply chain
Your system is your code plus thousands of dependencies, a build system, package registries and a CI pipeline. Each is a way in.
| Attack | How it works | Real example |
|---|---|---|
| Typosquatting / malicious packages | A look-alike package name hides malware | Recurring waves on PyPI and npm |
| Dependency confusion | A public package uses the same name as your internal one, with a higher version | Demonstrated against major companies in 2021 |
| Maintainer compromise | An attacker becomes or impersonates a trusted maintainer | xz utils backdoor (2024), after years of social engineering |
| Build-system compromise | Malicious code inserted during the build, not in the source | SolarWinds (2020) |
| CI pipeline abuse | Workflows expose secrets to untrusted code, such as pull requests from forks | Repeated incidents with CI misconfigurations |
Defenses:
- Lockfiles with hashes and pinned versions, so builds are reproducible and tampering is detected.
- Fewer dependencies; each one is trust you extend.
- SBOM (software bill of materials, in SPDX or CycloneDX format): an inventory of every component, so you can answer “are we affected?” in minutes when a new vulnerability lands.
- SLSA: a framework of increasing build-integrity levels, centred on signed provenance - a verifiable record of what was built, from which source, by which builder.
- Signing with Sigstore (Part 4).
- Hardened CI: minimal token permissions, third-party actions pinned to exact commits, and no secrets exposed to untrusted contributions.
AI link. ML projects pull enormous dependency trees. They also depend on models and datasets, which are part of the supply chain too (Part 12).
Detection and logging
Prevention will eventually fail somewhere. Time to detect then decides how much damage happens. Log what an investigator would need: logins, authorization failures, administrative actions, data access, outbound connections and - for AI systems - prompts, retrieved sources and tool calls. Centralize logs, protect them from tampering, and remember they are a data store too: logs full of prompts are personal data under GDPR.
Alert on anomalies: new access keys, logins from unusual places, sudden spikes in GPU or token spend.
MITRE ATT&CK catalogues real attacker behaviour as tactics (the attacker’s goal at each stage: initial access, execution, persistence, privilege escalation, defense evasion, credential access, discovery, lateral movement, collection, exfiltration, impact) and the techniques used for each. It gives defenders a shared vocabulary. MITRE ATLAS is its counterpart for AI systems (Part 8).
Incident response
The classic NIST lifecycle has four phases:
- Preparation - runbooks, contacts, and the practical ability to revoke keys and disable features quickly.
- Detection and analysis - confirm what happened and its scope.
- Containment, eradication, recovery - stop the bleeding, remove the cause, restore service.
- Post-incident activity - a blameless review that fixes the system, not the person.
Legal clocks start running too. Under GDPR, a personal-data breach must generally be reported to the supervisory authority within 72 hours of becoming aware of it. The EU’s NIS2 directive adds even shorter early-warning deadlines for organizations in its scope.
AI link. Prepare AI-specific levers in advance: roll back to a previous model version, disable a single agent tool, purge poisoned documents from a retrieval index, and revoke every token an agent holds.
Questions to ask
- What can each workload’s identity do, and what is the worst use of it if stolen?
- Is anything public that should not be - buckets, snapshots, dashboards?
- Could we list every dependency, model and dataset in production, with versions and sources?
- Who can change the CI pipeline, and which secrets can it reach?
- Which events are logged, who looks at them, and would we notice an attacker within a day?
- If an AI feature misbehaves at 2 a.m., how do we switch it off and roll it back?