7. Networking

A cloud network is a private network you define in software, called a VPC. You carve it into subnets and connect it to the internet only through doors you choose to open. Most security incidents, surprise bills and “why can’t X reach Y” problems live here.

The basics you need

• IP addresses. An IPv4 address such as 10.0.1.25 is 32 bits. Three ranges are reserved for private networks and never routed on the internet: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16.
• CIDR notation. The number after the slash is how many leading bits are fixed. 10.0.0.0/16 fixes 16 bits and leaves 16 free, so 65,536 addresses; /24 gives 256. Each extra bit halves the block.
• Ports. A port identifies a service on a machine: 443 for HTTPS, 22 for SSH, 5432 for Postgres, 6379 for Redis, 8000 or 8080 for most app servers.
• Protocols. TCP is reliable and ordered; UDP is fire-and-forget. On top of TCP: HTTP/1.1, HTTP/2, and gRPC (efficient service-to-service calls). LLM token streaming usually uses server-sent events (SSE) or WebSockets over HTTP.
• DNS turns names into IP addresses. Records are cached for their TTL, which is why DNS changes take time to spread.
• TLS encrypts traffic in transit using certificates. Clouds issue and renew them for free on their load balancers; Let’s Encrypt does the same elsewhere.
• NAT (network address translation) lets many private machines share one public address for outbound traffic.

The VPC and its doors

A VPC (virtual private cloud) is your isolated network inside one region, with an address range such as 10.0.0.0/16.

• Subnets are slices of that range, each inside one AZ. A public subnet has a route to the internet; a private subnet does not accept connections from the internet.
• Route tables decide where each packet goes next.
• An internet gateway is the two-way door for public subnets.
• A NAT gateway is a one-way door. Machines in private subnets can reach out (download packages, call APIs) but nobody can reach in.
• Security groups are stateful firewalls attached to each machine, listing what traffic is allowed. Network ACLs are stateless rules on a whole subnet. The principle for both: deny everything by default and open only what is needed.

The standard layout for an AI web service puts only the load balancer in public subnets. App servers, GPU servers and databases sit in private subnets across two or three AZs, reach the internet through NAT, and reach object storage through a private endpoint.

[embedded content: Standard VPC layout for an AI service · 3 subnet tiers]

Traffic enters only through the load balancer. Private machines reach out through NAT, and reach object storage through a free private endpoint instead.

A costly trap: NAT gateways charge for every GB that passes through them. Pulling large model weights or container images from S3 through NAT can cost more than the GPUs. A free S3 gateway endpoint inside the VPC avoids it.

Getting traffic in

• Load balancers spread requests over healthy backends and drop failed ones using health checks. A layer 4 (network) load balancer forwards TCP or UDP connections quickly without reading them. A layer 7 (application) load balancer understands HTTP, so it can route by path or hostname and terminate TLS.
• API gateways add authentication, rate limits and usage plans in front of your service.
• CDNs cache static content at edge locations near users, and usually add DDoS protection and a web application firewall.
• Reverse proxies such as nginx, Envoy, Caddy or Traefik do the same jobs on your own machines.

LLM-specific details:

• Timeouts. Streaming responses can last minutes. Load balancers often cut idle connections after 60 seconds by default, so raise it.
• Routing. Round-robin is a poor fit when requests differ hugely in length. Route by queue depth, and consider prefix-aware routing so requests sharing a long system prompt land where its KV cache already sits.

Private connectivity

• VPC peering connects two VPCs privately.
• Private endpoints (AWS PrivateLink, Google Private Service Connect, Azure Private Link) let you reach provider services, including model APIs such as Bedrock, without your traffic touching the public internet. Compliance reviews often require this.
• VPNs and dedicated lines (AWS Direct Connect, Google Cloud Interconnect, Azure ExpressRoute) connect cloud networks to offices and on-premises data centers.
• Transit hubs (AWS Transit Gateway and equivalents) connect many VPCs in a hub-and-spoke shape instead of a tangle of peerings.

Egress: the network bill

Data coming in is usually free. Data going out is not: on the big three, internet egress costs roughly $0.05–0.09 per GB, cross-region transfer a few cents per GB, and even cross-AZ traffic about $0.01 per GB in each direction on AWS.

• Serving 100 TB per month of downloads straight from object storage: about $5,000–9,000 per month.
• Copying a 50 TB dataset to another cloud once: about $2,500–4,500.

This is data gravity expressed in money. Mitigations: put a CDN in front, keep compute next to data, compress, or use storage with free egress such as Cloudflare R2. EU rules have pushed the big providers to waive egress fees when you move out entirely, but normal day-to-day egress is still billed.

High-performance networking for AI

• Two networks per GPU cluster. A “backend” network (InfiniBand, RoCE or AWS EFA) carries only GPU-to-GPU traffic. A “frontend” network handles storage, management and users.
• Topology. Large clusters use non-blocking fat-tree designs so any GPU can talk to any other at full speed. “Rail-optimized” means GPU number k in every server connects to the same switch group.
• Placement groups ask the provider to place your instances physically close together for the lowest latency and highest bandwidth.
• NCCL configuration. NVIDIA’s communication library picks the fastest available path. If it silently falls back to plain TCP sockets, training can run ten times slower. Setting NCCL_DEBUG=INFO shows which transport it chose.

Debugging toolkit

Work through connection problems in this order:

1. Does the name resolve? (dig, nslookup)
2. Is there a route? (traceroute, mtr)
3. Does a firewall or security group allow the port? (nc -zv host port)
4. Is the service listening on all interfaces (0.0.0.0) and not only 127.0.0.1? (ss -tlnp on the server). This is the most common bug of all.
5. Is TLS valid and is the response sane? (curl -v)
6. Is bandwidth what you expect? (iperf3)

Check yourself

• How many addresses are in a /20 block?
• Your GPU server in a private subnet cannot download a model from Hugging Face. Which two pieces must exist for it to work?
• Why might your bill show $3,000 for “NAT gateway data processing”?
• A streaming LLM endpoint always cuts off long answers at exactly 60 seconds. What is the most likely cause?