Serving and production patterns · Docker

In production a model container is a replaceable unit: configured by environment variables, honest about whether it’s ready, bounded by resource limits, observable through logs and metrics, and able to shut down without dropping requests. Those properties matter more than which server you pick.

Model servers you’ll meet

Server	Good for	Notes
vLLM	High-throughput LLM serving on GPUs	OpenAI-compatible API, continuous batching, Prometheus metrics
SGLang	LLM serving, structured and multi-call generation	Similar deployment model to vLLM
NVIDIA Triton Inference Server	Many frameworks at once (ONNX, TensorRT, PyTorch)	Dynamic batching, model ensembles, mature on NVIDIA hardware
Ollama, llama.cpp server	Local and edge inference with quantized models	Runs on CPU and natively on Apple Metal
Your own FastAPI service	Embeddings, classifiers, business logic around a model	Simplest to reason about; you own batching and concurrency

These projects release often and their flags change, so read the current docs and pin the image version you tested.

Readiness is not liveness

Loading a large model can take minutes, and during that time the container is “running” but useless. Give your service two endpoints: /health answers as soon as the process is up (liveness), /ready only once the model is loaded and a test inference succeeds (readiness). Docker’s HEALTHCHECK should probe readiness, with a start-period long enough for loading:

HEALTHCHECK --interval=30s --timeout=5s --start-period=300s --retries=3 \
  CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/ready', timeout=4)" || exit 1

Slim images don’t include curl, which is why this uses Python’s standard library. Compose’s condition: service_healthy and Kubernetes’ readiness, liveness and startup probes all build on the same distinction.

Configuration and secrets

Follow the twelve-factor rule: the same image runs in development, staging and production, and only the environment differs. Read settings such as MODEL_ID, MAX_BATCH_SIZE and LOG_LEVEL from environment variables. Secrets (API keys, database passwords, HF tokens) arrive at runtime from a secret manager, an untracked .env file, or Compose secrets mounted as files under /run/secrets/, never through the image.

Resource limits and threads

Set --memory and --cpus (or Compose’s deploy.resources.limits) so one runaway service can’t starve the host. Then align threads with the limit, because libraries size their thread pools from the host’s core count (Part 2): set OMP_NUM_THREADS and MKL_NUM_THREADS, and TOKENIZERS_PARALLELISM=false when a tokenizer runs inside forked workers. GPU memory is controlled by the server’s own settings, such as vLLM’s --gpu-memory-utilization.

Logs and metrics

Write logs to stdout, ideally as one JSON object per line with a request ID, and let the platform collect them; configure rotation (Part 5). Expose a Prometheus /metrics endpoint with request latency, tokens per second, queue length and GPU memory; vLLM and Triton provide one out of the box. For GPU-level metrics, NVIDIA’s DCGM exporter runs as its own container.

Graceful shutdown and zero-downtime deploys

A rolling update stops the old container, which means SIGTERM. Make sure your server receives it (exec form, Part 5), stops accepting new requests, and finishes in-flight ones within the grace period; long LLM generations may need stop_grace_period: 60s or more. For deploys without downtime on a single host: start the new container, wait until it’s ready, switch the reverse proxy (Caddy, Traefik or nginx) to it, then stop the old one.

A deployment ladder

One VM with Compose and a reverse proxy. Caddy or Traefik for TLS and routing, everything else on internal networks. Enough for demos, side projects and small production services.
Managed container platforms. Google Cloud Run, AWS ECS or Fargate, Azure Container Apps run your image without servers to manage; GPU-focused platforms such as Modal and RunPod do the same for inference and training. Check each one’s current GPU offering.
Kubernetes. EKS, GKE or AKS with the NVIDIA GPU Operator, plus KServe for model serving and KubeRay for distributed jobs, once you need many services, autoscaling and multi-tenant GPUs.