10. LLM and generative AI security

An LLM reads instructions and data as one stream of tokens, so any text it reads can act as an instruction. Prompt injection follows directly, and it still has no complete fix. Secure LLM applications are therefore designed to stay safe even when the model is fooled.

What you are actually securing

Everything the model sees is concatenated into one context window: the system prompt, the user’s messages, retrieved documents, tool results, memory. Role labels such as “system”, “user” and “tool” are conventions the model learned in training, not walls it cannot cross. Training models to prioritize higher-trust instructions (an “instruction hierarchy”) helps statistically. It does not create a hard boundary.

An LLM application has many parts to secure: the model, the orchestration code, the system prompt, the retrieval pipeline and vector store, tools, memory, the user interface that renders output, and logs.

Prompt injection

• Direct prompt injection - the user types instructions meant to override the developer’s (“ignore your previous instructions and…”). The attacker is the user.
• Indirect prompt injection - instructions hidden in content the model processes on the user’s behalf: a web page, an email, a PDF, a code comment, text inside an image, a calendar invite, a tool result. The attacker is a third party; the victim is the user, who never sees the payload. This is far more dangerous, and it was laid out systematically in a 2023 research paper by Greshake and colleagues.

Why it is unsolved. Recall Part 2. SQL injection was solved by giving queries a separate channel for data. LLMs have no such channel: “data” and “instructions” are both just text, and the model decides from context which is which. Filters are blocklists over an infinite language. Attackers rephrase, encode, switch languages, or hide instructions in images and audio. Detection classifiers lower the success rate, but not to zero, and an attacker can retry thousands of times.

Impact is set by capability, not by the injection. If the model can only produce text shown to the person who triggered it, the worst case is bad text. If it can read private data and take actions, injection becomes a confused-deputy attack (Part 5). This is why the most effective defenses limit what the model can do, rather than trying to perfect what it believes.

Jailbreaks are related but different

	Jailbreak	Prompt injection
Target	The model’s own safety policy	The application developer’s intended behaviour
Typical attacker	The user	A third party (indirect) or the user (direct)
Example	Getting a model to produce content its training forbids	A web page that makes an assistant email your files to a stranger
Main defenses	Safety training, input/output classifiers, monitoring	System design: least privilege, isolation, approvals, output handling

The techniques overlap: role-play framing, obfuscation, splitting a request across turns, many-shot prompts that fill a long context with fake dialogues, and optimization-based suffixes - strings found by gradient search that break safety training and often transfer between models.

Sensitive information disclosure

Leaks come from four paths: memorized training data (Part 9); retrieval returning documents the user may not see; secrets placed in the context; and data shared across users through caches, memory or logs. The single most important fix: authorize before retrieval. Filter what the user may see at query time, not by asking the model to keep quiet afterwards.

Hidden context exposure

Assume everything in the context window can be extracted: system prompt, tool schemas, retrieved policy text, configuration. Never put credentials there. Never rely on a hidden instruction as a security control. Leaked context also helps attackers write sharper injections.

Improper output handling

Model output flows into sinks: browsers, SQL, shells, file paths, templates, terminals, markdown renderers. Each needs the encoding and validation you would apply to any untrusted input.

One pattern deserves special attention: exfiltration through rendered images and links. An injected instruction tells the model to output a markdown image whose URL contains private data, such as an address on an attacker’s server with the user’s secrets appended. The chat interface automatically fetches the image to display it, and the data travels to the attacker. No click is needed. Variants of this have been found and fixed in many major AI products. Defenses: do not auto-load external resources, or proxy them through an allowlist; sanitize rendered markdown; set a strict content security policy.

Misinformation and overreliance

Hallucinations become security problems the moment someone acts on them. A well-known example: coding assistants invent plausible package names, and attackers register those names with malware inside (“slopsquatting”). In automated pipelines, a confident wrong answer becomes a wrong action that nobody reviewed. Defenses: ground answers in cited sources, verify before acting, and match human review to impact.

Unbounded consumption

The attacker’s cost is tiny; yours is large. Long outputs, reasoning modes, multimodal inputs and looping tool chains can make one request very expensive (“denial of wallet”). Defenses: clamp generation limits server-side (never let the client set them), budget by tokens per user and tenant, add circuit breakers that stop agent loops, and alert on spend anomalies. High query volume can also signal model extraction.

RAG and vector stores

A retrieval pipeline ingests documents, splits them into chunks, embeds them, stores the vectors, retrieves the nearest chunks for each query, and inserts them into the context. Risks:

• Poisoned documents. Anyone who can write into the corpus - a wiki, a shared drive, support tickets, the public web - can plant instructions or false facts. Research has shown that a handful of crafted passages can steer answers to targeted questions.
• Lost access control. Permissions attached to the original document disappear at chunking unless you carry them along.
• Cross-tenant leakage in shared indexes.
• Embedding inversion (Part 4).
• Retrieval manipulation - text crafted to be retrieved for many queries.

Defenses: store document permissions as chunk metadata and filter at query time; isolate indexes per tenant; record provenance for every chunk; validate what gets written; show users the sources behind each answer; and treat retrieved text as untrusted data.

The OWASP Top 10 for LLM Applications (2026 edition)

OWASP published the 2026 edition in August 2026. For the first time, the ranking was weighted with real incident data alongside the practitioner vote; Excessive Agency rose from sixth to third, and System Prompt Leakage was broadened into Hidden Context Exposure. Use the list as a shared vocabulary and checklist (official resource; summary of the changes).

ID	Risk	In one line	Covered in
LLM01	Prompt Injection	Untrusted text (or images, audio, video) steers the model	This part
LLM02	Sensitive Information Disclosure	Leaks via memorization, retrieval, or context	This part; Part 9
LLM03	Excessive Agency	More tools, permissions or autonomy than the task needs	Part 11
LLM04	Supply Chain	Models, adapters, libraries and runtimes from untrusted sources	Part 12
LLM05	Data and Model Poisoning	Corrupted training, fine-tuning data or adapters	Part 9
LLM06	Unbounded Consumption	Cheap requests trigger expensive compute	This part
LLM07	Misinformation	Wrong output acted on as fact	This part
LLM08	Hidden Context Exposure	System prompts, schemas and policies read back out	This part
LLM09	Vector and Embedding Weaknesses	Poisoned or leaky retrieval stores	This part
LLM10	Improper Output Handling	Model output reaching sinks unencoded	This part

Defense in depth for LLM applications

Start from the assumption that the model will be fooled sometimes, and ask what limits the damage. In rough order of effectiveness:

1. Limit reach. Give the model access only to the data and tools this task needs, scoped to this user.
2. Keep trusted instructions static. Build system prompts from fixed templates; never paste untrusted content into them.
3. Mark untrusted content. Delimit or encode retrieved text so the model can tell it is data. This lowers success rates but is not a boundary.
4. Constrain outputs. Use structured output schemas and allowlisted actions instead of free text wherever possible.
5. Secure every sink. Encode, sanitize, parameterize, and disable auto-fetching.
6. Classifiers on inputs and outputs for injections and policy violations, as one layer among several.
7. Human approval for consequential actions, showing the raw action rather than the model’s summary of it.
8. Monitor, log, rate-limit, and red-team continuously (Part 14).

Questions to ask

• What untrusted text can reach the context - directly, or via retrieval, tools, files and images?
• If the model obeyed an attacker perfectly, what is the worst it could do here?
• Is retrieval filtered by the user’s permissions before generation?
• Is anything in the context window secret?
• Where does model output go, is it encoded for that sink, and does any renderer fetch URLs automatically?
• What caps cost per request, per user and per agent loop?
• Was the injection defense tested against adaptive attackers, and what bypass rate remained?