2. The physical layer

Every cloud resource is a real machine in a real building in a specific place. That place decides your latency, which country’s law applies, what can fail together, and often the price.

Inside a data center

A data center is rows of racks. A rack is a steel frame about two meters tall holding servers, a top-of-rack network switch, and power distribution. Behind the racks sit the parts that matter most: grid connections, batteries and generators for outages, and cooling.

Typical machines you rent slices of:

• CPU server: two processor sockets with 64–192 cores in total, 0.5–2 TB of RAM, local NVMe SSDs, and a 25–200 Gb/s network card. One of these is cut into many VMs.
• 8-GPU AI server (for example an NVIDIA HGX H100 board): 8 GPUs, 2 CPUs, around 2 TB RAM, and eight 400 Gb/s network adapters just for GPU-to-GPU traffic. It draws about 10 kW, as much as a few households.
• Rack-scale AI system (for example NVIDIA GB200 NVL72): 72 GPUs wired together as one unit inside a single liquid-cooled rack, drawing around 120 kW.

Data centers are sized in megawatts of power, not square meters. Large AI campuses are now planned at hundreds of megawatts to gigawatts. Efficiency is measured by PUE (power usage effectiveness): total facility power divided by power reaching the servers. Modern facilities reach about 1.1–1.2.

Regions, availability zones and edge locations

Providers group data centers into a hierarchy designed around failure:

• Region - a geographic area such as Frankfurt (AWS calls it eu-central-1). Regions are isolated from each other, and most services exist separately in each region.
• Availability zone (AZ) - one or more data centers inside a region with independent power, cooling and networking. AZs sit kilometers apart so one fire, flood or grid failure takes out only one, yet close enough for a round trip of about 1–2 ms.
• Edge location - a smaller site near users, used for content caching (CDN), DNS and request routing. It does not run your general workloads.

The AZ is the unit of failure you design around. Run important services in two or three AZs and a whole building can go dark without taking you down. Run in two regions only when you need to survive a regional outage or serve distant users, because it adds cost and complexity.

Three practical details that trip people up:

• Some services are global. Identity (IAM), DNS and CDNs typically are; compute and storage are regional.
• AZ names can differ between accounts. On AWS, “eu-central-1a” in your account may be a different building than in mine; use AZ IDs when coordinating.
• GPUs are not spread evenly. New GPU types launch in a few regions first, and a specific AZ can simply run out (AWS reports InsufficientInstanceCapacity). For AI work, region choice is often decided by where the GPUs are.

For you in Germany: every major provider has a Frankfurt region, and keeping personal data in the EU makes GDPR compliance simpler (Part 8). GPU supply may be better in other EU regions, so check before you commit.

Latency numbers worth memorizing

These orders of magnitude let you sanity-check any design in your head.

Operation	Approximate time
CPU L1 cache reference	1 ns
Main memory (DRAM) reference	100 ns
Read 4 KB from a local NVMe SSD	20–100 µs
Network round trip inside one AZ	0.1–0.5 ms
Round trip between AZs in one region	0.5–2 ms
First byte of a small object from object storage (S3)	20–200 ms
Round trip Frankfurt ↔ London	10–15 ms
Round trip Frankfurt ↔ US East	80–90 ms
Round trip Frankfurt ↔ US West	140–160 ms
Time to first token from a hosted LLM API	0.2–2 s
Generating 500 output tokens from a hosted LLM	5–20 s

The lesson for AI apps: one model call usually dwarfs network latency. But an agent that makes 30 sequential calls pays every hop 30 times, so placement matters again.

Bandwidth numbers worth memorizing

Bandwidth is how much data moves per second; latency is how long the first byte takes. AI systems are usually limited by bandwidth.

Link	Approximate bandwidth
GPU high-bandwidth memory (HBM)	3–8 TB/s per GPU
NVLink, GPU to GPU inside a server or rack	0.9–1.8 TB/s per GPU
PCIe Gen5 x16, CPU to GPU	~64 GB/s each direction
Local NVMe SSD	3–14 GB/s per drive
GPU server network, all adapters together	~400 GB/s (8 × 400 Gb/s)
Standard cloud VM network	1–12 GB/s (10–100 Gb/s)
Home internet	0.01–0.1 GB/s

Two things to notice. First, networks are quoted in bits (Gb/s) and storage in bytes (GB/s); divide bits by 8, and many estimation mistakes are exactly this. Second, there is roughly a thousandfold gap between a GPU’s own memory and the network. Almost all of distributed AI engineering is about working around that gap.

Check yourself

• Why does running in two AZs protect you from a power failure but not from a bad deployment?
• How long does it take to pull a 140 GB model from object storage over a 10 Gb/s link? What changes at 100 Gb/s?
• A 30-step agent in Frankfurt calls a model hosted in Virginia. How much of its total time is pure network?
• Why would a provider put a new GPU type in only three regions at launch?