Same price, 50% more expensive: the hidden cost of renting GPUs

If you want to train an AI model, you rent GPUs (graphics chips, the workhorse processors that do the heavy maths behind modern AI) by the hour from a cloud provider. The number everyone shops on is the GPU-hour price: so many dollars per chip per hour. It looks like the one figure you need to compare two providers.

SemiAnalysis (an independent semiconductor and AI-infrastructure research firm) argues that number is close to meaningless on its own. In their analysis, two clusters advertising the identical GPU-hour price can differ by 10%, 15%, or even 50% in what the same workload actually costs, once the hidden charges are counted. They sort those charges into seven buckets: storage, network, the control plane (the software that schedules and manages the machines), support, goodput, setup, and debugging.

The most important of those, and the least visible, is goodput.

Goodput is a plain idea wearing a technical word. You pay for every GPU-hour the meter runs. Goodput is the fraction of those paid hours that produced useful work, rather than being thrown away when something broke mid-job and the work had to be redone.

Here is the analogy. Picture two cars advertised at the same price per kilometre. One drives smoothly to work every morning. The other stalls halfway, every single commute, so you coast back home and start the trip over. The sticker price per kilometre is identical. The real cost of actually arriving is not, because you keep paying for distance that got you nowhere. A GPU cluster that fails mid-training and forces a restart is that second car.

The stalling is not a rare fault. It is what large clusters do. SemiAnalysis notes that when roughly 4,000 GPUs are wired together into a single training system, the cluster fails on average every 6.25 hours. Thousands of chips, plus the networking and cooling around them, add up to so many parts that at any given moment something is statistically likely to be failing. A training run that takes weeks therefore hits interruption after interruption.

Each failure means falling back to the last saved checkpoint and recomputing whatever was lost. That recomputed time is billed at the same GPU-hour rate as the useful work, which is exactly why goodput, not the sticker, decides the bill.

The gap between providers is large and measured. SemiAnalysis puts wasted-work loss at 6.14% for their top Gold-tier providers, 10.53% for the big hyperscalers (the giant general-purpose clouds), and 20.91% for weaker Silver-tier providers. Same advertised price, and one in five paid hours quietly evaporates on the worst tier.

The underlying idea is not. Anyone who has run large computing systems knows that reliability and total cost of ownership matter more than a headline unit price, and telecoms and data centres have argued about effective versus advertised cost for decades. What is new is that AI training has pushed cluster sizes so high, and GPU hours so expensive, that a few percent of wasted work now translates into very real money on a single run.

To make the comparison legible, SemiAnalysis built ClusterMAX, a rating system that scores more than 80 GPU clouds across performance, networking, storage, security, support and pricing. It is one firm's methodology, not an industry standard, so the exact tier boundaries and percentages are their measurements rather than settled fact. The direction of the finding, though, is hard to argue with: the sticker is not the cost.

For anyone renting serious GPU capacity, the practical lesson is to stop comparing GPU-hour rates as if they were the whole story and start asking what fraction of those hours will survive to do useful work. A cheaper-looking cluster that loses 20% to failures can cost more than a pricier one that loses 6%.

For the rest of us, this is a useful reminder about how AI economics actually work underneath the model demos. The compute that trains these systems is not a smooth, reliable utility. It is a large fleet of failure-prone machines, and a meaningful slice of the money spent goes into simply recovering from things breaking. When you next see a striking figure for what a model cost to train, remember that some of that number bought no progress at all. It only paid to redo work the hardware had already lost.