16. Evaluation, failure modes and security

Benchmarks tell you roughly how capable a model is; only an evaluation built from your own task tells you whether your system works. And because an LLM receives instructions and data through the same channel - text in its context - any untrusted text it reads can try to take control. That single fact drives most LLM security problems.

Intrinsic evaluation

Perplexity measures how well a model predicts held-out text. It’s useful for comparing training runs, but only with the same tokenizer and evaluation data: different tokenizers split text differently, so per-token numbers aren’t comparable. Bits per byte fixes that. Low perplexity doesn’t guarantee helpfulness.

Benchmarks and their problems

Benchmarks test knowledge (e.g., MMLU), math (GSM8K, MATH), coding (HumanEval, SWE-bench), expert reasoning (GPQA) and more. Read their numbers with care:

• Contamination: test questions leak into web-scale training data and inflate scores.
• Saturation: top models score near the ceiling, so differences become noise.
• Prompt sensitivity: formatting, few-shot examples and answer extraction can move scores by several points.
• Goodhart’s law: once a benchmark becomes a target, it gets optimized for.
• LLM-as-judge biases: judges favour longer answers, the option shown first and their own style. Validate any judge against human labels.
• Preference leaderboards: pairwise human votes capture what users like, which includes style, not just correctness.

Evaluating your own system

1. Collect real (or realistic) inputs and define what “good” means for each case.
2. Score with code where possible (exact match, schema validity, tests), with rubric-guided LLM judges where not, and with humans on a sample.
3. Evaluate the system - retrieval, prompts, tools, post-processing - not just the model.
4. Run the set on every change (prompt, model version, chunking) as regression tests.
5. Track cost and latency next to quality.

Why models hallucinate

From first principles: the model is trained to produce plausible continuations, and nothing in next-token pretraining rewards saying “I don’t know” - a fluent guess often scores better than a hedge. Hallucinations cluster around rare facts, precise details (numbers, citations, URLs) and questions with false premises. Post-training can reduce or worsen this, depending on what was rewarded.

Mitigations: ground answers in retrieved sources with citations, ask for abstention when the sources don’t cover the question, verify claims with tools or a second pass, constrain outputs to known options, and prefer models trained to abstain.

Other failure modes worth naming

• Sycophancy: agreeing with the user’s stated beliefs.
• Reversal curse: a model trained on “A is B” may not infer “B is A”.
• Long-context degradation: missed information in long inputs, especially mid-context.
• Brittleness to rephrasing: small prompt changes flip answers.
• Non-determinism: outputs vary across runs, even at low temperature.
• Compounding errors in agents: small per-step error rates multiply over many steps (Part 17).

Security: the threat model

The core issue resembles SQL injection before parameterized queries: there is no reliable separation between instructions and data in an LLM’s context.

Threat	What happens	Main defences
Direct prompt injection / jailbreak	the user crafts input to override instructions or safety training	safety training, input/output classifiers; assume system prompts can leak
Indirect prompt injection	instructions hidden in web pages, emails, documents or tool outputs that the model reads	treat all retrieved content as untrusted; least-privilege tools; human confirmation for consequential actions
Data exfiltration	injected instructions make the model leak data through a tool call, a link or a rendered image URL	restrict outbound actions and URLs; don’t auto-render untrusted links or images; keep sensitive data apart from untrusted inputs
Excessive agency	an agent with broad permissions is steered into harmful actions	minimal permissions, sandboxing, approval gates, audit logs
Training-data extraction	the model regurgitates memorized personal or secret data	deduplication, filtering, never training on secrets
Data poisoning / backdoors	malicious training or fine-tuning data plants hidden behaviours	data provenance, vetting, evaluating for triggers
Supply chain	malicious model files (e.g., Python pickle files that run code when loaded), compromised dependencies	load safetensors; verify sources and hashes
Model extraction	an attacker distills your model through its API	rate limits, monitoring, terms of use

The OWASP Top 10 for LLM Applications is a good checklist. Design rule: assume the model can be manipulated by anything it reads, and limit what a manipulated model could do.

Questions you should now be able to answer

• Why can’t you compare perplexity across models with different tokenizers, and what metric fixes that?
• A model scores 90% on a public benchmark but 60% on your task. List three explanations.
• Explain from the training objective why models invent citations.
• Your agent reads emails and can send emails. Describe an indirect prompt injection attack and three defences.
• Why is loading an untrusted .pt file risky, and what’s the alternative?