14. Debugging and introspection

Debug a container by asking four questions in order: did it start, what did it print, what does it see from inside, and which limit or permission stopped it. Each question has a command, and most failures are answered by the second one.

The toolkit

Command	Question it answers
docker ps -a	Is it running? If not, what was its exit code and when?
docker logs --tail 200 -t <name>	What did it print before it died?
docker inspect <name>	Its full config: env, mounts, networks, IP, State.OOMKilled, State.ExitCode
docker inspect -f '{{json .Mounts}}' <name>	One field at a time, using Go templates
docker exec -it <name> sh	What does the filesystem, environment and network look like from inside?
docker run --rm -it --entrypoint sh <image>	Explore an image whose normal command crashes on start
docker stats	Live CPU, memory, network and disk use per container
docker top <name>	Which processes run inside it
docker events	Stream of starts, stops, OOM kills and health changes
docker cp <name>:/path ./local	Copy files out of (or into) a container, even a stopped one
docker history, dive	Which layer made the image big, and what’s in it
docker buildx build --progress=plain .	Full, unabbreviated build output
docker debug <name>	A toolbox shell in slim images that lack one (Docker Desktop)
nsenter -t <pid> -n	On Linux, enter just the container’s network namespace with host tools like tcpdump

Symptom, cause, fix

Symptom	Likely cause	Fix
Exit 137, OOMKilled: true	Memory limit or host RAM exceeded	Raise the limit, shrink batch size, look for leaks; on Mac also check Docker Desktop’s VM memory
Exit 137, OOMKilled: false	SIGKILL after the stop timeout, or external kill	Handle SIGTERM, use exec form, lengthen the grace period
Exit 139	Segfault in native code	Mismatched torch, CUDA or extension builds; rebuild them together
“exec format error”	Wrong CPU architecture, usually an arm64 image from a Mac on an amd64 server	docker buildx build --platform linux/amd64
“could not select device driver with capabilities: [[gpu]]”	NVIDIA Container Toolkit missing or not registered	Install it, run nvidia-ctk runtime configure, restart Docker
“CUDA driver version is insufficient”	Host driver older than the image’s CUDA	Upgrade the driver, or use an older CUDA build
“no kernel image is available for execution on the device”	Binary not compiled for this GPU architecture	Newer torch or CUDA build, or set TORCH_CUDA_ARCH_LIST when compiling
torch.cuda.is_available() is False	No --gpus, a CPU-only torch wheel, or running on the Mac	Run nvidia-smi in the container, print torch.version.cuda
DataLoader “bus error” or worker killed	64 MB /dev/shm	--shm-size=16g or --ipc=host
“Connection refused” from the host	App bound to 127.0.0.1, or the wrong port mapped	Bind 0.0.0.0; check docker port <name>
One container can’t reach another via localhost	Separate network namespaces	Use the service name on a shared network
“Permission denied” on a bind mount	UID mismatch, or SELinux labels on Fedora/RHEL hosts	-u $(id -u):$(id -g), fix ownership, add :z to the mount on SELinux
“No space left on device”	Images, build cache, volumes or unrotated logs	docker system df, prune, configure log rotation
Every build reinstalls all packages	Cache busted by an early COPY . .	Copy lockfiles first, add .dockerignore
Logs show up late or in bursts	Python output buffering	PYTHONUNBUFFERED=1
Container restarts endlessly	Crash on start under restart: always	docker logs, then run the image with --entrypoint sh to investigate

A debugging habit

When something fails, write down which layer you think it lives in before typing a command: image, build, runtime, network, storage, GPU stack, or your code. Then pick the command that tests that guess. This is the fluency the guide is aiming at: each error message points to one component, and you learn to read which.