14. Cloud economics

Cloud cost is price times quantity, divided by how much of it you actually use. In AI systems the GPU and token lines dominate, and the cheapest GPU-hour is the one you did not leave running.

How you pay

Pricing model	Typical discount vs on-demand	Commitment	Risk	Fits
On-demand	None	None	GPU capacity may simply be unavailable	Experiments, unpredictable load
Spot / preemptible	About 50–90%	None	Reclaimed with 30 seconds to 2 minutes’ notice	Checkpointed training, batch inference, CI
Reserved, committed use, savings plans	About 30–70%	1–3 years of spend	You pay even if you stop using it	A steady baseline you are sure of
Capacity reservations or blocks	Varies	A fixed time window	You pay for the whole window	Planned training runs
GPU cloud term contracts	Negotiated	Months to years	Lock-in and supplier risk	Sustained large GPU use

Rough GPU prices for orientation, to be checked against current offers: H100s on GPU clouds and marketplaces have commonly rented for around $2–3.50 per GPU-hour; hyperscaler on-demand prices have often been noticeably higher, with commitments closing much of the gap. Older or smaller GPUs (A100, L40S, L4) cost proportionally less, and consumer cards on marketplaces less again.

Effective cost is price divided by utilization

A $2.50 H100 that is busy 25% of the time really costs $10 per useful hour. The levers that raise utilization are batching, queues in front of GPUs, autoscaling, sharing development GPUs, scheduling jobs back to back, and switching things off.

Where surprise bills come from

• Idle resources: forgotten GPU instances, notebooks left running over the weekend, unattached disks, old snapshots, load balancers with no traffic.
• Data transfer: internet egress, chatty cross-AZ traffic, and NAT gateway processing fees (Part 7).
• Storage growth: checkpoints and logs that are never deleted, and versioned buckets quietly keeping every old copy.
• Observability ingestion: verbose logs and traces at scale can cost more than the service they describe.
• Small fixed fees that add up: support plans, public IPv4 addresses (AWS charges for each), managed-platform surcharges.
• Token spend: long prompts resent on every call, no caching, and agents looping without step limits.

Controls (FinOps practice)

• Budgets and alerts on day one, before your first GPU launch, plus anomaly detection.
• Tags or labels on every resource (project, owner, environment), so every euro can be attributed.
• Automatic shutdown for idle development GPUs and notebooks.
• Lifecycle policies on every bucket that grows.
• Quotas as a hard safety net against runaway scripts or leaked keys.
• Unit economics: cost per 1,000 requests, per active user, or per document processed. These are the numbers a business decides on.
• A monthly review of the top five line items.

Token economics for API-based systems

Output tokens usually cost several times more than input tokens, and “thinking” tokens from reasoning models are billed as output. Cached input is billed at a steep discount.

Worked example with illustrative prices of $3 per million input tokens and $15 per million output tokens: a RAG answer with 6,000 input and 500 output tokens costs $0.018 + $0.0075 = about $0.026. At 100,000 answers a month that is about $2,550. If prompt caching halves the effective input cost, it falls to about $1,650.

The levers, roughly in order of effort: shorter prompts and tighter retrieval, prompt caching for stable prefixes, capping output length, routing easy requests to a smaller model, using batch APIs for anything not urgent, and step limits on agents.

Total cost of ownership

For small teams, engineering time is usually the largest cost. A self-managed Kubernetes GPU cluster can easily cost more in salaries than in compute. Slow iteration has an opportunity cost too. A sound default: use managed services and APIs until a specific, measured cost or constraint justifies taking on the work yourself.

Learning without a big bill

• Free GPU notebooks (Google Colab, Kaggle) cover early experiments.
• GPU marketplaces (RunPod, Vast.ai) and serverless GPU platforms (Modal, which includes a small monthly free allowance) are the cheapest way to get real GPU hours.
• The big three offer free tiers and startup credit programs (AWS Activate, Google for Startups Cloud Program, Microsoft for Startups), worth applying to if you build a product.
• Set a budget alert before launching anything, every single time.

Check yourself

• Your team runs 4 H100s around the clock for inference at 30% utilization. Name three ways to cut cost without hurting latency.
• Which is cheaper per useful hour: a $2 GPU at 30% utilization or a $3 GPU at 60%?
• Which three line items would you check first on a surprisingly high monthly bill?
• Estimate the monthly API cost of an app idea of your own using the token formula.