Math for AI from First Principles

All the math behind modern AI fits into eight pillars. Each one exists because a model forces a specific question, and each pillar is the answer humanity worked out.

Overview: eight pillars, one model

Pillar	The question it answers	Where you meet it in AI
0. Language of math	How do I state and read a precise claim?	Every paper, every formula, every API doc with math in it
1. Linear algebra	How is information represented and transformed?	Embeddings, weight matrices, attention, LoRA, PCA
2. Calculus	How does the output change when I nudge an input or a parameter?	Gradients, backpropagation, autodiff
3. Probability	How do I reason precisely about uncertainty?	Softmax outputs, next-token sampling, diffusion, Bayesian ideas
4. Statistics	What can I honestly conclude from finite data?	Training as estimation, overfitting, evals, A/B tests
5. Information theory	How much surprise or information is in something?	Cross-entropy loss, perplexity, KL penalties in RLHF and distillation
6. Optimization	How do I find parameters that make the loss small?	SGD, Adam, learning-rate schedules, weight decay
7. Numerical and systems math	What happens on real hardware with finite precision, memory and money?	bf16, quantization, FLOPs, KV-cache size, scaling laws

[embedded content: pillar map · 8 pillars and the capstone]

Read it top to bottom: each arrow means “learn this first”. Optimization pulls gradients from calculus and its loss from statistics and information theory, and Pillar 7 applies everything on real hardware.

The first-principles method

First principles means you never accept a formula you cannot rebuild. For every concept in this guide, run the same six questions until each has an answer you could say out loud.

1. Why does this exist? What problem was impossible or painful before this idea?
2. What exactly is it? State the definition in one precise sentence, then in symbols.
3. What is the smallest example? Compute a 2-dimensional or 3-number case by hand.
4. What does it look like? Find the geometric or physical picture behind the symbols.
5. Where does it break? Name the assumption that, if false, makes the idea fail.
6. Where is it in a model? Point to the line of code or the layer where it lives.

What fluency means here

Fluency is five abilities, not memorized formulas. You can read the notation aloud in plain words. You can compute a tiny example on paper.

You can explain why the idea works, not only what it does. You can implement it in a few lines of NumPy, which doubles as Python practice. You can spot it in a paper or a codebase and say what would change if it were removed.

How each pillar is laid out

Every pillar section follows the same order so you always know where you are:

• Why it exists: the problem that forced the idea.
• Core ideas: each concept built from the one before, with the intuition first and the formula second.
• In AI: exactly where it appears in real models and systems.
• Questions to ask: the questions that separate surface knowledge from real understanding.
• Fluency check: small tasks that prove you own it.

Read the pillars in order the first time. After that, use the question banks as a map for revisiting any topic.