3. Virtualization: how one machine becomes many

Virtualization is the trick that turns one physical server into many isolated, rentable pieces. Virtual machines, containers and serverless are three points on one trade-off: stronger isolation and control on one end, faster start-up and less overhead on the other.

Virtual machines

A hypervisor is software that sits between the hardware and several guest operating systems, giving each one the illusion of its own computer. Modern CPUs have built-in support for this (Intel VT-x, AMD-V), so a VM runs at close to native speed. Cloud hypervisors run directly on the hardware (KVM, Xen, Hyper-V); desktop ones such as VirtualBox run on top of an OS.

Large providers offload networking, storage and security to dedicated cards. AWS calls its version Nitro. This leaves almost all CPU and memory for customers and keeps the hypervisor tiny.

Things to understand about cloud VMs:

• A vCPU is usually one hardware thread, not one core. With hyperthreading, 8 vCPUs are typically 4 physical cores. Expect performance accordingly.
• Instance types encode a recipe. In AWS m7i.2xlarge: m = general purpose family, 7 = generation, i = Intel, 2xlarge = 8 vCPUs and 32 GB RAM. Families are tuned for a ratio: general (m), compute-heavy (c), memory-heavy (r), storage-heavy (i), accelerated (p and g for NVIDIA GPUs, trn and inf for AWS’s own AI chips).
• Images and disks. A VM boots from an image (AWS calls it an AMI) onto a network-attached boot disk. Stopping a VM keeps the disk and stops compute billing; terminating deletes it. Local instance storage is wiped when the VM stops.
• GPU VMs get the real GPU. The physical GPU is passed straight through to your VM over PCIe, so performance is close to bare metal.
• Neighbors share hardware. A busy tenant on the same host can affect cache and memory bandwidth (the “noisy neighbor” problem). Dedicated hosts and bare-metal instances exist for strict isolation or licensing.

Containers

A container is not a small VM. It is an ordinary Linux process that the kernel fences off using two features:

• Namespaces control what the process can see: its own process list, network interfaces, filesystem mounts, hostname and users.
• Control groups (cgroups) control what it can use: CPU share, memory limit, I/O.

All containers on a host share one kernel. That is why they start in under a second and add almost no overhead, and also why their isolation is weaker than a VM’s.

An image is a stack of read-only filesystem layers plus metadata (the start command, environment variables). It is built from a Dockerfile, stored in a registry (Docker Hub, GitHub Container Registry, AWS ECR, Google Artifact Registry) and identified by a tag (myapp:1.4) or an exact content hash (sha256:...). Pin the hash when you need reproducibility, because tags can be moved.

Why containers matter so much for AI: the image carries every user-space dependency, including the exact Python, CUDA libraries and PyTorch version. The environment that worked on your laptop or in CI is byte-for-byte what runs on the GPU server.

GPUs inside containers

The split of responsibilities is the source of most GPU container bugs:

• The host runs the NVIDIA kernel driver. It is not inside your container.
• The container carries the CUDA user-space libraries (CUDA runtime, cuDNN, NCCL) and your framework.
• The NVIDIA Container Toolkit exposes the GPU devices and driver into the container at start.

The rule: the host driver must be new enough for the CUDA version in your image. Newer drivers run older CUDA; older drivers refuse newer CUDA with an error like “CUDA driver version is insufficient.” When a GPU container fails on start, check this first.

AI images are large: a PyTorch plus CUDA image is often 5–15 GB, and pulling it is part of start-up time. Model weights usually stay out of the image and are loaded from object storage or a cache volume. Baking them into the image is a valid trade when start-up speed matters more than image size.

Serverless

Serverless means you hand over code or a container, and the platform runs it on demand. It scales from zero to many copies and bills only while requests are running. There are still servers; you just never see or manage them.

Two main forms:

• Functions (AWS Lambda, Google Cloud Run functions, Azure Functions): short, event-triggered pieces of code.
• Serverless containers (Google Cloud Run, AWS Fargate, Azure Container Apps): any container that listens on a port.

Triggers can be an HTTP request, a message on a queue, a file landing in a bucket, or a schedule. This makes serverless natural glue for event-driven pipelines, such as “when a PDF is uploaded, extract text and embed it.”

The central cost of serverless is the cold start. When no copy is warm, a request waits while the platform finds capacity, pulls the image, starts the runtime and loads your code. For a small CPU function that is 0.1 to a few seconds. For a GPU service that must load a 20 GB model it can be tens of seconds or more. The fixes are keeping a minimum number of warm copies (which you pay for), smaller images, faster model loading, and memory snapshots where the platform supports them.

Serverless GPU platforms (Modal, RunPod Serverless, Replicate, Baseten, and Google Cloud Run with GPUs) apply this model to accelerators. They fit spiky inference and batch jobs where paying for an idle GPU 24/7 would be wasteful.

Service models and shared responsibility

The classic labels are IaaS (infrastructure: VMs, disks, networks), PaaS (platform: you bring code), SaaS (software: you use a finished product), and FaaS (functions). The labels matter less than the line they draw between your job and the provider’s.

Layer	VM (IaaS)	Managed Kubernetes	Serverless	Model API (SaaS)
Your data and who can access it	You	You	You	You
Application code	You	You	You	Provider
Runtime and dependencies	You	You (the image)	You (the image)	Provider
Operating system patching	You	Shared	Provider	Provider
Scaling	You	Shared	Provider	Provider
Hardware, buildings, hypervisor	Provider	Provider	Provider	Provider

This is the shared responsibility model. The provider secures the cloud itself; you secure what you put in it. The top row never moves: your data and access settings are always your job, and that is where most real breaches happen.

Check yourself

• A teammate says a container is “a lightweight VM.” What is wrong with that, and when does the difference matter?
• A GPU container fails with “CUDA driver version is insufficient.” What is mismatched, and which side do you change?
• Why do serverless platforms that run untrusted code wrap each container in a tiny VM?
• When would you bake model weights into the image instead of downloading them at start?