GPU/AI workload spend has gone from roughly 4% to 18% of enterprise cloud budgets since 2023 — and most engineering teams still can't answer a simple question when finance asks: why does this cost what it costs? The honest answer is that GPU cost isn't one decision, it's a chain of six decisions, and a mistake anywhere in that chain shows up as an unexplainable number at the end.

Step 1: Match the GPU to the workload's actual VRAM and bandwidth needs - not the biggest chip you can get.

The instinct is to reach for the most powerful GPU available "to be safe." But most cost waste here isn't compute — it's VRAM and memory bandwidth sitting idle because the model or batch size never comes close to using what you provisioned. A model that fits comfortably in 24GB of VRAM doesn't get faster on an 80GB card; it just costs more per hour for headroom nobody's using. Size to the model's actual memory footprint first, and treat raw compute (FLOPs) as the second-order decision.

Step 2: Fix scheduling and batching before you touch autoscaling.

This is the step most teams skip straight past, and it's usually the single biggest lever. A GPU running requests one at a time, with gaps between them, can sit at 20-30% utilization while still billing for 100% of the hour. Continuous/dynamic batching — grouping incoming requests so the GPU processes them together instead of sequentially — routinely gets utilization into the 70-90% range on the exact same hardware. If you're reaching for more GPUs before you've fixed batching, you're buying capacity to compensate for a scheduling problem, not a real capacity shortage.

Step 3: Get autoscaling and multi-tenancy right, or you're paying for idle GPUs twice over.

GPUs don't scale up and down as cleanly as CPU-based services — cold starts are slower, and provisioning is often coarser-grained. Two things matter here: autoscaling policies tuned to GPU-specific cold-start times (a policy copied from your CPU services will either scale too late or thrash), and multi-tenancy where it's safe — sharing a GPU across multiple smaller workloads (via MIG partitioning or similar) instead of dedicating a full card to something that uses a fraction of it. Skipping multi-tenancy is one of the most common ways teams end up with a fleet of GPUs each running at 15% utilization.

Step 4: Track cost per output — tokens, inferences, whatever your actual unit is — not cost per GPU-hour.

This is the step that actually closes the loop back to "why does this cost what it costs." Cost-per-GPU-hour tells you what you're spending. Cost-per-token (or cost-per-inference) tells you whether you're spending it well — and it's the only metric that lets you compare a batching fix, a scheduling fix, or a model-size change on equal footing. Without this, teams "optimize" by staring at GPU utilization dashboards without any way to connect a utilization improvement to an actual dollar-per-output number finance can understand. Set this up before capacity planning gets hard, not after.

The actual takeaway: GPU cost problems almost never live where teams look first (which GPU, how many). They live in the middle of the chain — scheduling, batching, and multi-tenancy — where utilization actually gets determined. Fix the middle before you touch the ends, and track cost-per-output the whole way through so you can prove it worked.

Wrestling with a specific GPU cost breakdown — utilization, batching, or a bill that doesn't add up? Reply to this email and I'll help you think through it.