AI chips are not short of math. They are short of memory bandwidth

Ask most people what makes an AI chip fast and they will say something about calculations per second. The chipmakers encourage this: Nvidia's headline number for its Rubin chip, announced at its GTC conference in March, is 50 petaflops (50 million billion arithmetic operations per second) in the compact FP4 number format used for running models. That is real, and it matters for training. But for the thing most of us actually use, an AI model answering a question, the chip spends most of its time not calculating. It is waiting for data to arrive from memory.

The reason is how a model produces text. A language model is a very large table of numbers, called weights. A big frontier model today has on the order of a trillion of them, which at four bits each is about 500 gigabytes. To produce one word (strictly, one token, a word fragment), the chip has to run the input through the whole table: read every weight, multiply, add. Then it does it again for the next token. Every single token requires reading the entire model out of memory.

The arithmetic per weight is tiny, a multiply and an add. The reading is the hard part. So the question "how many tokens per second can this chip generate for one user" is, to a first approximation, "how many times per second can it read 500 GB". That is memory bandwidth, measured in terabytes per second. A chip with 8 TB/s can read a 500 GB model about 16 times a second; at 22 TB/s, about 44 times. Real systems complicate this (they serve many users at once so each read is shared, and they use tricks so not every weight is touched), but the ceiling is set by bandwidth, not by math. The engineers' term for this is being memory-bound.

Once you know to look for it, the last year of chip announcements reads differently. Nvidia's Rubin keeps the same 288 GB of memory as the Blackwell Ultra chip it replaces, but reads it at up to 22 TB/s, using a new generation of stacked memory called HBM4. Blackwell reads its memory at 8 TB/s. That is a 2.75x jump in bandwidth, slightly more than the roughly 2.5x jump in FP4 compute on Nvidia's own figures. For the first time in several generations, the memory pipe grew faster than the calculator.

Amazon's Trainium3, which SemiAnalysis dissected in a report we covered yesterday, tells the same story from the budget end of the market. Capacity went from 96 GB to 144 GB per chip. Bandwidth went up about 70%, from 2.9 to roughly 4.9 TB/s, by running the memory's pins at 9.6 gigabits per second instead of 5.7 and switching suppliers to get there. Amazon doubled the chip's arithmetic in the same generation, but the memory gain is the one SemiAnalysis flagged as the meaningful change for the inference jobs Anthropic will run on it.

The clearest evidence came in our brief last Thursday. SemiAnalysis reported that Nvidia's Rubin Ultra, the chip after Rubin, will ship with 192 GB of memory rather than the 1,024 GB originally planned, but with the same bandwidth, by using shorter memory stacks that keep every data wire while dropping the extra layers. Nvidia has not confirmed the change. If it is right, Nvidia is choosing to cut capacity by four fifths to protect bandwidth, because the memory makers cannot produce enough HBM4 wafers and bandwidth is what the customer is paying for. SemiAnalysis's phrase for the trade was "almost a free lunch". The same supply squeeze is why it expects the first Rubin shipments to land closer to 20 TB/s than the advertised 22.

It also explains a company that is otherwise hard to explain: SK Hynix, a South Korean memory maker that most people had never heard of, became one of the most profitable chip companies in the world by making most of the HBM stacked next to Nvidia's chips. When the scarce resource is bandwidth, the company that makes bandwidth wins.

No, and the age of the idea is the point. In 1995 two computer scientists, William Wulf and Sally McKee, published a short paper titled "Hitting the Memory Wall", observing that processor speed was improving around 50% a year while memory speed improved around 7%, so eventually a processor would spend all its time waiting. A line usually attributed to the supercomputer designer Seymour Cray makes the same point more bluntly: anyone can build a fast CPU; the trick is to build a fast system. The whole history of chip design since then is a series of workarounds for the wall: caches, prefetching, and finally HBM, which stacks memory chips vertically right next to the processor to shorten the wires. HBM was introduced by SK Hynix and AMD in 2013 and first shipped in a graphics card in 2015.

What is new is the workload. Video games and scientific simulations were often compute-bound; the calculator was the bottleneck and the memory kept up. Generating text from a trillion-parameter model is the most memory-hungry mainstream workload ever built, and it is now the most valuable one. The wall did not move. The world's most expensive software walked straight into it.

Two caveats. Training a model, as opposed to running it, is still heavily compute-bound, which is why the petaflop numbers are not a con. And bandwidth is not the only limit on serving: the wires between chips (NVLink and its rivals) and the memory used to hold each conversation's context both matter, and were the subject of earlier briefs. But if you want one number to judge an inference chip by, it is the TB/s figure, and it is usually in the second paragraph of the press release, not the first.

Picture a restaurant kitchen with one star chef who can plate a dish in ten seconds. On paper the kitchen can serve 360 dishes an hour. In practice the ingredients live in a cold room down the corridor, and one runner can carry enough for about 100 dishes an hour. The chef spends most of the shift standing at an empty counter. Hiring a chef who plates in five seconds does nothing; the kitchen still serves 100 an hour. Doubling the runners, or moving the cold room next to the stove, doubles the output.

HBM is the cold room moved next to the stove. Bandwidth is the number of runners. The petaflop number is how fast the chef's hands are, and for this particular restaurant the chef has been waiting on the runners for years. Rubin, in this picture, roughly triples the runners while making the chef only somewhat faster. Rubin Ultra, if the report is right, makes the cold room smaller so it can afford the same number of runners.

If you want one sentence: an AI chip answering you is limited by how fast it can read its own memory, not by how fast it can do arithmetic, and the chipmakers' spec sheets show them quietly racing on that number.

For anyone using these tools, the practical consequence is where the speed-ups come from. When your chatbot gets noticeably faster next year, it will mostly be because of new memory, not new math. And when someone tells you a chip is a certain number of times faster, ask which number they mean.

Three things to watch. Whether Nvidia confirms the Rubin Ultra memory cut, which would be the plainest statement yet that bandwidth beats capacity. Whether SK Hynix, Samsung and Micron can ramp HBM4 fast enough to hit the advertised 22 TB/s in volume. And the results of Anthropic's bet on Trainium, since Amazon's whole pitch is that a chip with less math and adequate bandwidth is the cheap way to serve tokens.