Linear algebra · Math for AI

Linear algebra is the data structure of AI: every input, weight, activation and embedding is a vector, matrix or tensor. If you master one pillar deeply, make it this one.

Why it exists

Real data is many numbers at once: a pixel grid, a word’s meaning, a user’s history. We need to treat those many numbers as one object and transform them in a predictable way.

“Linear” means the transformation respects addition and scaling: f(a·x + b·y) = a·f(x) + b·f(y). Linear problems are the ones mathematics can solve completely. So deep learning builds complex functions from linear pieces glued together with simple nonlinearities.

Core ideas

1. Vectors: two views of the same thing

The computer view: a vector is a list of numbers, x = (x₁, …, xₙ). The geometric view: it is an arrow from the origin, or a point in n-dimensional space.

You need both views at once. The list is what the code stores; the geometry is what gives the operations meaning. When people say “similar words have nearby embeddings”, they are using the geometric view.

A vector space is any set where you can add elements and scale them by numbers, and the usual rules hold. ℝⁿ is the main example, but functions and polynomials form vector spaces too.

2. Linear combinations, span, basis, dimension

A linear combination is a weighted sum: a₁v₁ + a₂v₂ + … + aₖvₖ. The span of some vectors is everything you can reach with such sums.

Vectors are linearly independent if none of them is a combination of the others: each one adds a genuinely new direction. A basis is a set of independent vectors that spans the whole space. The number of vectors in any basis is the dimension.

Coordinates are only meaningful relative to a basis. The same arrow has different numbers in a different basis. This is why individual embedding dimensions rarely mean anything on their own: the model never promised to use a human-readable basis.

3. Dot product, length, angle, projection

The dot product multiplies matching entries and adds them up. Its geometric meaning is what makes it powerful:

x \cdot y = \sum_{i=1}^{n} x_i y_i = \|x\|\,\|y\|\cos\theta

It is large when two vectors point the same way, zero when they are perpendicular (orthogonal), and negative when they point apart. Dividing out the lengths gives cosine similarity, which compares direction only.

The length (L2 norm) is ‖x‖ = √(x·x). Other norms measure size differently: L1 sums absolute values and encourages sparsity, L∞ takes the largest entry.

The projection of x onto a direction u (with ‖u‖ = 1) is (x·u)u: the shadow x casts on that line. Projection is the geometric heart of least squares, PCA and attention.

4. Matrices are functions, not tables of numbers

This is the single most important idea in the pillar. A matrix A is a linear map, and its columns tell you everything: column j is where the j-th basis vector lands.

So multiplying A by x means “take x₁ of column 1, plus x₂ of column 2, and so on”:

Ax = x_1 a_1 + x_2 a_2 + \dots + x_n a_n

Matrix multiplication AB is composition: first apply B, then A. That is why it is associative, (AB)C = A(BC), but not commutative: rotating then stretching differs from stretching then rotating.

The shape rule is the type system: (m×n) times (n×p) gives (m×p), and the inner sizes must match. The cost is m·n·p multiply-adds, about 2mnp floating-point operations (FLOPs). That number drives the cost of every model you will run.

Learn four views of the same product AB, because papers switch between them without warning:

Entry (i, j) is the dot product of row i of A with column j of B.
Each column of AB is A applied to a column of B.
Each row of AB is a combination of the rows of B.
AB is a sum of outer products: column k of A times row k of B, summed over k.

5. Solving Ax = b: what can be reached, what is lost

The column space of A is every output A can produce. The null space is every input that A sends to zero: information A destroys.

The rank is the dimension of the column space: how many independent directions survive. The rank–nullity theorem says rank + dimension of null space = number of columns. Whatever is not kept is lost.

If A is square and full rank, it has an inverse and Ax = b has exactly one solution. In practice you almost never compute A⁻¹; you call a solver, which is faster and more numerically stable.

When there is no exact solution, you find the closest one. Least squares picks x so that Ax is the projection of b onto the column space, which gives the normal equations:

A^\top A\,\hat{x} = A^\top b

That is linear regression, derived from geometry alone.

6. Determinant: how much a map scales volume

The determinant of a square matrix is the factor by which it scales area or volume. A negative sign means orientation flips, like a mirror.

A determinant of zero means the map squashes space into a lower dimension. That is exactly when information is lost and no inverse exists. You will rarely compute determinants by hand, but the volume picture explains normalizing flows and change-of-variables in probability.

7. Eigenvectors and eigenvalues: the natural axes of a map

Most vectors get rotated and stretched by A. Eigenvectors are the special directions that only get stretched:

Av = \lambda v

The eigenvalue λ is the stretch factor. Their power shows up under repetition: applying A k times multiplies each eigen-direction by λᵏ. Directions with |λ| > 1 explode and those with |λ| < 1 vanish.

That is the mathematical root of exploding and vanishing gradients in deep and recurrent networks. It also explains why residual connections and careful initialization matter.

Symmetric matrices (A = Aᵀ) are the friendly case. The spectral theorem says they have real eigenvalues and perpendicular eigenvectors, so they are pure stretches along orthogonal axes. Covariance matrices and Hessians are symmetric, which is why this case appears everywhere.

8. Singular value decomposition: every matrix, explained

Every matrix, even a non-square one, factors into rotate, stretch, rotate:

A = U \Sigma V^\top

V rotates the input, Σ stretches along the axes by the singular values σ₁ ≥ σ₂ ≥ … ≥ 0, and U rotates into the output space. The number of nonzero singular values is the rank.

Keeping only the top k singular values gives the best possible rank-k approximation (the Eckart–Young theorem). Large singular values carry most of what the matrix does; small ones are often noise. This one fact underlies compression, denoising, PCA and LoRA.

9. Positive definite matrices: bowls

A symmetric matrix A is positive definite if xᵀAx > 0 for every nonzero x. Geometrically, the function xᵀAx is a bowl with a single lowest point.

Covariance matrices are positive semidefinite (≥ 0). A Hessian that is positive definite at a point means you are at a local minimum (Pillar 2 and Pillar 6).

10. Tensors as used in ML

In ML code, a tensor just means a multi-dimensional array: a batch of sequences of embeddings has shape (batch, sequence, d_model). Most of the math applies along one or two axes while the others ride along.

Two tools make this manageable. Broadcasting stretches smaller arrays to match larger ones. Einstein summation (einsum) names every axis and states which ones are summed, which is the clearest way to write attention.

In AI

Embeddings: each token becomes a row of an embedding matrix, a point in ℝ^d where direction carries meaning.
Linear layers: y = Wx + b is the workhorse; an MLP block is two of them around a nonlinearity.
Attention: every piece is linear algebra. QKᵀ computes all pairwise dot products between queries and keys at once.
Retrieval and RAG: vector search ranks documents by cosine similarity to a query embedding.
LoRA fine-tuning: the weight update is forced to be low rank, ΔW = BA, exploiting the SVD insight.
PCA and visualization: projecting embeddings onto the top singular directions to see structure.

The attention formula, the most important line in modern AI, reads as pure linear algebra plus one softmax:

\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

Why divide by √d_k? If query and key entries are independent with mean 0 and variance 1, their dot product has variance d_k. Dividing by √d_k brings the variance back to 1, so softmax does not saturate.

And why does LoRA save so much? A full update to a 4096×4096 matrix trains about 16.8 million numbers. A rank-8 update, B (4096×8) times A (8×4096), trains 65,536: about 256 times fewer.

Questions to ask

What space does this vector live in, and does any single coordinate mean anything?
What does this matrix do geometrically: rotate, stretch, project, or collapse?
What is its rank, and what information does it destroy?
Is this step linear? If not, where exactly is the nonlinearity, and why is it needed?
What are the shapes, and what is the multiply-add cost?
Why use cosine similarity here instead of the raw dot product?
Would the result change if I rotated the basis? If not, coordinates are not meaningful.
What do the largest singular values or eigenvalues tell me about this matrix?

Fluency check

Multiply a 2×2 matrix by a vector by hand, once by rows and once as a combination of columns.
Write the 2D rotation matrix for 90° and explain each column.
Find the eigenvalues and eigenvectors of [[2, 1], [1, 2]] by hand (answer: 3 and 1, along (1, 1) and (1, −1)).
Derive the normal equations from the projection picture.
In NumPy, take the SVD of a 100×100 random matrix, rebuild it with rank k, and plot error against k.
Implement single-head attention in NumPy with an einsum, writing the shape of every array.