GPU Training Cost Calculator
A single H100 training run can quietly cost more than a senior engineer’s monthly salary, and most teams only find out after the invoice arrives. The hourly rate is the smallest part of the story — the real spend is driven by how many GPUs you reserve, how long they run, whether you chose spot or on-demand, and the storage and egress you forgot to model. This calculator totals all of it and compares your run across seven cloud providers so you can see whether AWS’s premium is worth it or whether RunPod and Lambda Labs will do the same job for a third of the price.
Configure Your Training Run
The GPU Training Cost Formula
The true cost of a training run is not the GPU sticker price — it is the sum of compute, persistent storage, and data movement:
Total = (GPU_Hourly × GPUs × Hours × ModeFactor × (1 + Markup)) + (Storage_GB × $0.10) + (Egress_GB × $0.09)
The mode factor is 1.0 for on-demand and roughly 0.35 for spot. Multiply by a managed-platform markup only if you are using SageMaker, Vertex AI, or Azure ML, which wrap the raw GPU price with orchestration, logging, and a convenience tax of 10–30%. Teams that run on bare-metal clouds (Lambda Labs, RunPod, CoreWeave) or their own colocated hardware skip that markup entirely.
Choosing the Right GPU: It Is About Memory, Not Just Speed
The single most common training mistake is renting an H100 when an A100 would have finished the job for half the price, or conversely renting an A10G that runs out of memory and silently spills to host RAM, slowing the run 10x. The deciding constraint is almost always GPU memory: model weights, optimizer states, and activations must fit with headroom. A 7B model in full precision needs ~28GB just for weights plus optimizer states; an 80GB A100 handles it comfortably, while a 24GB card forces quantization or activation checkpointing.
- T4 / L4 (16–24GB): inference and light fine-tuning only. Never for pre-training.
- A10G (24GB): the budget workhorse for small models and diffusion inference.
- A100 40/80GB: the default for 7–70B fine-tuning and mid-size pre-training. The 80GB variant matters when batch size or context length is large.
- H100 / H200: justified only when wall-clock time is the constraint or when you need the extra memory/throughput for 70B+ work. Otherwise the premium rarely pays back.
Worked Example: Fine-Tuning Llama-3-8B on 8×A100
Suppose you fine-tune Llama-3-8B across 8 A100 80GB GPUs for 24 hours, on-demand, with 500 GB of dataset and checkpoints stored and 200 GB of data egressed to your app. Compute is 8 × $1.80 × 24 = $345.60. Storage adds 500 × $0.10 = $50, and egress adds 200 × $0.09 = $18. The total is about $413.60. Flip to spot pricing and compute drops to 8 × $0.65 × 24 = $124.80, bringing the total to roughly $192.80 — a 53% saving, provided your training loop checkpoints and can resume after a preemption. Run the same 8×H100 job on-demand and compute alone jumps to $614.40, a reminder that the GPU choice, not the hours, is usually the dominant cost lever.
Frequently Asked Questions
How much does it cost to train a model on an A100 or H100?
On-demand, an A100 80GB is ~$1.80/GPU-hour and an H100 ~$3.20. Eight A100s for 24 hours is about $345 in compute before storage and egress; eight H100s about $614. Spot cuts those 55–65%.
Is spot pricing worth the risk for training?
Yes for checkpointed, resumable, or repeatable jobs where 55–65% savings matter. No for a one-shot multi-day run with no resume point — use on-demand or reserved instead.
Why is the H100 more expensive than the A100?
Higher throughput (FP8, faster interconnect) shortens wall-clock time for large models. The premium pays back only for compute-bound, well-distributed workloads — not memory- or I/O-bound ones.
What hidden costs inflate a GPU bill?
Data egress (~$0.09/GB), idle storage accrual, and managed-platform markup (10–30% on SageMaker/Vertex). A $400 compute job can become $700 all-in.
How many GPUs do I need?
7–8B fine-tuning fits on a single A100 80GB or a quantized A10G; 70B training needs dozens. Memory for weights + optimizer + activations must fit with headroom — 8×A100 is the standard pod for 7–13B work.
Buy or rent GPUs?
If utilization exceeds ~60% of the year, buying and colocating can break even in 12–18 months. For intermittent or experimental work, cloud spot/reserved is cheaper and far more flexible.