7. Numerical and systems math

The first six pillars assume perfect numbers and free computers. This pillar is what happens on real hardware, with finite precision, finite memory and a budget. For AI systems work, it is where most day-to-day decisions are made.

Why it exists

Computers store numbers with a fixed number of bits, so most real numbers are rounded. Hardware moves data far more slowly than it computes on it. Ignoring either fact gives you NaN losses, slow inference, and cost estimates that are wrong by 10×.

Core ideas

1. Floating point: scientific notation in binary

A floating-point number is stored as sign × mantissa × 2^exponent. The exponent bits set the range, from tiny to huge. The mantissa bits set the precision: how many significant digits survive.

Format	Bits (sign / exponent / mantissa)	Largest value	Precision step near 1	Typical use
fp32	1 / 8 / 23	about 3.4 × 10³⁸	about 1.2 × 10⁻⁷	master weights, optimizer state
bf16	1 / 8 / 7	about 3.4 × 10³⁸	about 7.8 × 10⁻³	training and inference default
fp16	1 / 5 / 10	65,504	about 9.8 × 10⁻⁴	older mixed-precision training
fp8 (E4M3 / E5M2)	1 / 4 / 3 or 1 / 5 / 2	small	coarse	fast matmuls on newer hardware

bf16 keeps fp32’s range but drops precision, so values rarely overflow. That is why it replaced fp16 for training. fp16 overflows past 65,504, which forced tricks like loss scaling.

Rounding has surprising consequences. In fp32, 1 + 10⁻⁸ equals exactly 1. Addition is not associative, so summing the same numbers in a different order gives slightly different results: one root of GPU nondeterminism.

Subtracting two nearly equal numbers wipes out most significant digits (catastrophic cancellation). The fix is always the same: rearrange the formula so you never need that subtraction.

2. Stable formulas: never compute what you can’t store

Softmax overflows if any logit is large, because e^1000 is infinite in any format. Subtracting the maximum logit first changes nothing mathematically but keeps every exponent at most 0. The same idea gives the log-sum-exp trick:

The product of 1,000 probabilities of 0.1 is 10⁻¹⁰⁰⁰, which underflows to zero. The sum of their logs, about −2,303, is perfectly fine. So models work with log-probabilities, and libraries fuse softmax with cross-entropy into one stable operation.

3. Conditioning: how much errors get amplified

A problem is ill-conditioned when small input errors cause large output errors. For a matrix, the condition number κ = σ_max / σ_min (Pillar 1) measures this. Roughly, you lose log₁₀ κ decimal digits of accuracy.

This is why you solve linear systems instead of inverting matrices, and why normalization layers like LayerNorm and RMSNorm help. They keep activations in a well-behaved range where floating point is accurate.

4. Quantization: fewer bits per number

Quantization maps floats onto a small integer grid with a scale s and a zero point z:

The rounding error is at most half a grid step, so the scale matters. One large outlier stretches the grid and wastes precision for everything else. That is why practical schemes use a separate scale per channel or per small group of weights.

Moving weights from 16 bits to 4 bits cuts their memory by 4×. Whether accuracy survives depends on outliers and on which layers you quantize.

5. Counting FLOPs

A matrix multiply of (m×n) by (n×p) costs about 2mnp FLOPs (Pillar 1). For a dense transformer with N parameters, two rules of thumb cover most estimates.

• Forward pass: about 2N FLOPs per token, since every parameter is used in one multiply-add.
• Training: about 6N FLOPs per token, because the backward pass costs about twice the forward pass.

So total training compute is roughly:

A worked example: 7 billion parameters on 2 trillion tokens is about 8.4 × 10²² FLOPs. On GPUs sustaining an assumed 400 TFLOP/s each, that is about 58,000 GPU-hours. Change any assumption and the arithmetic tells you the new cost in seconds.

6. Memory math

Weights take parameters × bytes per parameter. A 7B model is about 14 GB in bf16, 7 GB in int8 and 3.5 GB in int4.

Training needs far more. With Adam in mixed precision, a common accounting is about 16 bytes per parameter before activations. That covers bf16 weights and gradients, an fp32 master copy, and Adam’s two fp32 moments. A 7B model then needs about 112 GB, which is why training state is sharded across GPUs.

For inference, the KV cache stores each past token’s keys and values in every layer:

Take a typical 8B-class configuration: 32 layers, 8 KV heads, head size 128, bf16. That is 128 KiB per token, or 1 GiB for one 8,192-token sequence. With 32 KV heads instead of 8, it would be 4× larger; that saving is the whole point of grouped-query attention.

7. Arithmetic intensity: compute-bound or memory-bound?

Arithmetic intensity is FLOPs performed per byte moved from memory. GPUs can compute far faster than they can read memory. So an operation with low intensity waits on memory, no matter how fast the arithmetic units are.

This explains the two phases of LLM inference. Prefill processes the whole prompt in parallel: large matrix multiplies, compute-bound. Decoding generates one token at a time and must read every weight for each token: memory-bound.

That gives a quick ceiling. A 14 GB model on hardware with an assumed 3 TB/s of memory bandwidth needs at least about 4.7 ms per token at batch size 1, so at most roughly 200 tokens per second. Batching many users reuses each weight read, which is why serving systems batch aggressively.

8. Complexity and scaling with sequence length

Big-O notation describes how cost grows with input size. Self-attention compares every token with every other, so its compute grows as O(n²) in sequence length n, while the MLP layers grow as O(n). Long context is expensive for this reason.

IO-aware kernels such as FlashAttention compute exact attention without ever storing the full n×n matrix in slow memory. The FLOPs stay similar, but memory traffic drops sharply: Pillar 7 thinking turned into speed.

9. Power laws and scaling laws

A power law y = a·x^(−k) is a straight line on a log–log plot, with slope −k. Language-model loss falls roughly as a power law in parameters, data and compute, down toward the irreducible entropy from Pillar 5.

Fitting this curve tells you how to split a compute budget between model size and data. The widely cited compute-optimal rule of thumb is about 20 training tokens per parameter. Many recent models train well past that, trading extra training compute for a smaller model that is cheaper to serve.

10. Orders of magnitude

Systems thinking runs on powers of ten. Know the scales: KB, MB, GB, TB for memory; GFLOP/s, TFLOP/s, PFLOP/s for compute; nanoseconds to milliseconds for latency. A quick Fermi estimate, right to within 2×, beats a precise answer to the wrong question.

In AI

• Choosing a GPU or a model size: weights + KV cache + activations must fit in memory, with headroom.
• Serving cost: tokens per second follow from bandwidth (decode) and FLOPs (prefill); price per token follows from that.
• Training stability: NaNs and loss spikes are often numerical: overflow, a bad scale, or an unstable formula.
• Quantization decisions: memory saved versus accuracy lost, measured with the statistics from Pillar 4.
• Architecture choices like grouped-query attention, mixture-of-experts and long-context methods are mostly answers to memory and FLOP budgets.

Questions to ask

• What precision is this in, and what are its largest value and its precision step?
• Where could this overflow or underflow? Am I working in log space?
• Is this operation compute-bound or memory-bound?
• How many FLOPs and how many bytes per token does this take?
• What dominates memory here: weights, optimizer state, activations, or KV cache?
• How does cost grow with sequence length, batch size and model size?
• What do we lose by quantizing, and where are the outliers?

Fluency check

• In NumPy, show that np.float32(1) + np.float32(1e-8) equals 1, and that summation order changes a result.
• Implement softmax naively and with max-subtraction, then feed both logits of 1,000.
• Compute the weight memory of a 70B model in bf16, int8 and int4 (140, 70 and 35 GB).
• Read an open model’s config file and compute its KV cache per token and per full context.
• Estimate training FLOPs and GPU-hours for a model of your choice with C ≈ 6ND.
• Estimate the decode speed ceiling of a model from its size and your GPU’s memory bandwidth.
• Plot a power law on linear axes, then on log–log axes, and read off the exponent.