10. GPUs in containers

To a container, an NVIDIA GPU is a set of device files (/dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm) plus driver libraries that must exactly match the host’s kernel driver. The NVIDIA Container Toolkit injects both when the container starts. So the image carries CUDA libraries but never the driver, and one image runs on any host whose driver is new enough.

The four pieces and who owns them

The NVIDIA Container Toolkit

Install the NVIDIA driver and the Container Toolkit on the host, then register it with Docker: sudo nvidia-ctk runtime configure --runtime=docker and restart the daemon. From then on, --gpus all tells Docker to have the toolkit mount the GPU devices, the matching driver libraries and nvidia-smi into the container.

The smoke test that proves the whole chain works:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Choosing GPUs: --gpus all, --gpus 2 (any two), or --gpus '"device=0,2"' (specific ones; the nested quotes are needed). Inside the container, the selected GPUs are renumbered from 0. Newer setups can also use the Container Device Interface (CDI), a vendor-neutral spec file describing devices, with --device nvidia.com/gpu=all.

CUDA compatibility, decoded

• The “CUDA Version” in nvidia-smi is a ceiling, not what’s installed: it’s the newest CUDA runtime that this driver supports. A container with CUDA 12.1 on a driver showing 12.4 is perfectly fine.
• Newer driver, older CUDA: works. Older driver, newer CUDA: fails with “CUDA driver version is insufficient for CUDA runtime version”. Within one major CUDA version, NVIDIA’s minor-version compatibility relaxes this somewhat; check NVIDIA’s CUDA compatibility page for exact driver minimums.
• PyTorch wheels bring their own CUDA runtime (the cu121, cu124, cu128 builds). So a plain python:3.12-slim base plus a CUDA PyTorch wheel runs on GPU; you don’t need an nvidia/cuda base. You need a CUDA devel image only to compile CUDA code: custom kernels, or extensions like flash-attn built from source.
• New GPU, old build: “no kernel image is available for execution on the device” means the binary wasn’t compiled for your GPU’s architecture. Very new GPUs (for example the Blackwell generation) need recent CUDA and PyTorch builds.

Base image flavours

Shared memory

Docker gives each container only 64 MB of /dev/shm. PyTorch’s DataLoader with num_workers > 0 moves batches between processes through shared memory, so it fails with a “bus error” or “DataLoader worker killed” in Docker even though the same code works on bare metal. NCCL also uses shared memory for GPU-to-GPU traffic within a node. Fix it with --shm-size=16g or --ipc=host; model servers such as vLLM document one of these as required.

Multi-GPU and multi-node

On one machine, --gpus all plus torchrun --nproc_per_node=<gpus> inside the container works like bare metal, provided shared memory is large enough. Across machines, add host networking and InfiniBand access (Part 8), and usually let a scheduler (Slurm with Pyxis and Enroot, or Kubernetes with the NVIDIA GPU Operator) handle placement.

Docker cannot limit GPU memory: --memory caps system RAM only. Two containers on the same GPU can push each other out of memory. Real isolation comes from Multi-Instance GPU (MIG) on A100 and H100 class cards, which splits a GPU into hardware-isolated slices that appear as separate devices, or from framework settings such as vLLM’s --gpu-memory-utilization.

AMD GPUs

AMD’s ROCm stack works similarly but without a toolkit layer: pass the devices directly with --device /dev/kfd --device /dev/dri and use ROCm images such as rocm/pytorch.

On your Mac

Containers in Docker Desktop run in a Linux VM with no access to the Apple GPU, so CUDA is unavailable and PyTorch’s Metal backend (MPS) doesn’t work inside containers; everything runs on CPU. The working pattern: run GPU-hungry local inference natively (Ollama, llama.cpp, MLX use Metal) and call it from containers through host.docker.internal. Docker Desktop’s Model Runner feature takes the same approach, running models on the host and exposing an API to containers.

For CUDA work, rent a Linux GPU machine and drive it from your Mac’s terminal with a Docker context over SSH:

docker context create gpu-box --docker "host=ssh://ulvi@<server-address>"
docker context use gpu-box
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
docker context use default   # back to the Mac's engine

Every docker command now runs on the remote engine, including builds, which then happen natively on amd64.