10. Infrastructure as code and delivery

Infrastructure as code (IaC) means your cloud setup is written in text files that are versioned and reviewed like software. It can then be recreated exactly, audited, and torn down completely. Use the console to explore; use code for anything you intend to keep.

Why code instead of clicks

Clicking resources into existence leaves no record of what was done or why. It cannot be reviewed before it happens, cannot be repeated in another region, and drifts as people make small manual changes.

Code fixes all of that. Changes show up as diffs in pull requests, the same definition builds dev, staging and production, and one command can destroy everything at the end of an experiment. That last point matters enormously with GPUs, where a forgotten instance costs real money every hour.

Terraform and OpenTofu from first principles

Terraform is the most widely used IaC tool. OpenTofu is its open-source fork, created after HashiCorp changed Terraform’s license in 2023, and works almost identically.

• Declarative. You describe the resources you want; the tool works out how to get there.
• Providers are plugins that know how to talk to AWS, Google Cloud, Azure, Cloudflare, Kubernetes and hundreds of other APIs.
• State is Terraform’s record linking each block of your code to a real resource ID. Keep it in remote storage with locking (for example an S3 bucket), never only on a laptop and never in git, since it can contain secrets.
• Plan, then apply. terraform plan shows the exact difference between your code and reality; terraform apply executes it. Read every plan, especially lines that say a resource will be destroyed and recreated.
• Modules are reusable bundles, such as “a VPC with private subnets” or “a GPU node pool.”
• Drift is when reality no longer matches the code because someone changed something by hand. A plan reveals it.

A small example: a bucket for datasets and one L40S GPU machine in a private subnet with its own role.

resource "aws_s3_bucket" "datasets" {
  bucket = "acme-ml-datasets-eu"
}

resource "aws_instance" "trainer" {
  ami                  = var.deep_learning_ami_id
  instance_type        = "g6e.xlarge"            # 1 x NVIDIA L40S
  subnet_id            = aws_subnet.private_a.id
  iam_instance_profile = aws_iam_instance_profile.trainer.name
  tags = { project = "finetune-demo" }
}

Alternatives follow the same ideas: Pulumi lets you write infrastructure in Python or TypeScript; AWS CDK and CloudFormation, and Azure Bicep, are provider-specific; Crossplane manages cloud resources through Kubernetes.

CI/CD

• Continuous integration (CI): every push runs tests and linters, builds the container image, and scans it for known vulnerabilities.
• Continuous delivery (CD): the tested artifact is deployed automatically to staging, then to production behind approvals or checks.
• Tools: GitHub Actions and GitLab CI are the most common. They authenticate to the cloud through OIDC roles, not stored keys (Part 8).

Deployment strategies, from simplest to safest:

• Rolling: replace old copies with new ones a few at a time.
• Blue-green: run the new version beside the old one and switch all traffic at once, with instant rollback.
• Canary: send a small share of traffic, say 5%, to the new version and watch error rates and latency before going further.
• Shadow: the new model receives a copy of real traffic, but its answers are only logged and compared, never returned. Ideal for comparing a new model safely.

GitOps applies the reconciliation idea to deployment. A Git repository holds the desired state of the cluster, and a controller (Argo CD or Flux) keeps the cluster matching it. To deploy, you merge a pull request.

Reproducibility for ML

An ML result is reproducible only if every input is pinned:

• Container base image, by digest.
• Python packages, through a lock file (uv, Poetry).
• CUDA and framework versions, plus the minimum driver they need.
• Model revisions, by commit hash on the Hugging Face Hub, not just the model name.
• Dataset versions, as immutable paths or hashes.
• Configs and random seeds, logged with each run in an experiment tracker such as MLflow or Weights & Biases.

Build once, deploy many: the exact image tested in staging is the one promoted to production, never a rebuild. For multi-step work (prepare data, train, evaluate, register, deploy), pipeline tools such as Airflow, Dagster, Prefect, Kubeflow Pipelines or Metaflow run each step in order and record what happened.

Evaluations as tests

For LLM applications, classic unit tests are not enough. Keep a fixed set of inputs with graders (exact match, rubrics, or another model acting as judge), and run it in CI on every change to a prompt, model or retrieval setting. Block the deployment when quality drops. This one habit prevents most silent regressions.

Check yourself

• What is Terraform state, and what goes wrong if it lives only on your laptop?
• A plan says your production database will be destroyed and recreated. What do you do next?
• How does a canary deployment differ from a shadow deployment, and when would you choose each for a new model?
• Name six things you must pin to reproduce a fine-tuning run six months from now.