18. Glossary

The terms you will hear most, alphabetically, each with the part that explains it.

Term	Meaning	Part
All-reduce	Every GPU contributes a tensor and all receive the sum; used to average gradients	11
ANN index	Approximate nearest neighbor index (HNSW, IVF) that makes vector search fast at some cost in recall	6
Arithmetic intensity	FLOPs performed per byte moved; decides whether work is memory-bound or compute-bound	4
Availability zone (AZ)	Independent data center(s) within a region; the basic unit of failure	2
BF16	2-byte number format with FP32’s range; the default for training and inference	4
Blast radius	How much breaks when one thing fails or one change goes wrong	13
Canary deployment	Sending a small share of traffic to a new version before all of it	10
CIDR	Notation for IP address ranges; /16 means 16 fixed bits	7
Cold start	Delay while a new instance provisions, pulls its image and loads its model	3, 12
Continuous batching	Requests join and leave the running batch at every decoding step	12
Control plane	The APIs that create, change and permission resources	8
CUDA	NVIDIA’s GPU programming platform and library ecosystem	4
Data gravity	Large data is slow and costly to move, so compute moves to it	1, 7
DCGM	NVIDIA’s GPU monitoring toolkit and metrics exporter	13
Egress	Data leaving a cloud, region or zone; usually billed per GB	7
Envelope encryption	Data encrypted with a data key, which is itself encrypted by a master key in a KMS	8
Error budget	The failure your SLO permits: 100% minus the target	13
FinOps	The practice of managing and attributing cloud cost	14
FSDP / ZeRO	Sharding weights, gradients and optimizer state across data-parallel GPUs	11
Gang scheduling	Starting all parts of a distributed job together or none at all	9
Goodput	The share of time or requests that produced useful results meeting targets	11, 12
HBM	High-bandwidth memory stacked beside a GPU; its main memory	4
Hypervisor	Software that runs multiple VMs on one physical machine	3
IaC	Infrastructure as code: cloud resources defined in versioned files	10
IAM	Identity and access management: who may do what on which resource	8
Idempotency	Doing an operation twice has the same effect as doing it once	13
InfiniBand / RoCE / EFA	High-bandwidth, low-latency networks with RDMA for GPU clusters	4, 7
KV cache	Stored attention keys and values for past tokens during generation	5, 12
Least privilege	Granting only the permissions a task needs	8
LoRA / QLoRA	Fine-tuning small adapter matrices on a frozen (optionally 4-bit) base model	5
MFU	Model FLOPs utilization: achieved FLOPs divided by peak	5, 11
MIG	Hardware partitioning of one GPU into isolated slices	4
NAT gateway	Lets private machines make outbound connections; billed per GB	7
NCCL	NVIDIA’s library for collective communication between GPUs	11
Neocloud	A GPU-focused cloud provider such as CoreWeave, Lambda or Nebius	15
NVLink / NVSwitch	NVIDIA’s direct high-speed GPU-to-GPU links inside a server or rack	4
Object storage	Whole files stored by key behind an HTTP API, such as S3	6
OIDC federation	Letting an outside identity (such as a CI job) assume a cloud role without stored keys	8
PagedAttention	Storing the KV cache in fixed-size blocks to avoid fragmentation	12
Prefill / decode	Processing the prompt in parallel (compute-bound) vs generating tokens one by one (memory-bound)	5, 12
Prefix caching	Reusing the KV cache of a shared prompt prefix	12
RDMA / GPUDirect	Network transfers straight into another machine’s (or GPU’s) memory without the CPU	4
Reconciliation loop	A controller that keeps moving reality toward declared desired state	9
Region	A geographic group of availability zones	2
Roofline model	Performance is capped by either compute or memory bandwidth, depending on intensity	4
RPO / RTO	How much data you can lose / how long recovery may take	13
SLI / SLO / SLA	What you measure / your target / your contractual promise	13
Speculative decoding	A cheap draft proposes tokens that the large model verifies in one pass	12
Spot instance	Spare capacity at a deep discount that can be reclaimed at short notice	11, 14
Static stability	Staying up during an outage without needing to create new resources	8
Taint / toleration	Kubernetes marks that keep ordinary pods off special nodes such as GPUs	9
Tensor parallelism	Splitting individual weight matrices across GPUs; needs NVLink-class links	11
TTFT / TPOT	Time to first token / time per output token	12
vCPU	One hardware thread, usually half a physical core	3
VPC	Your isolated, software-defined private network in a region	7