GPU

Neural networks reduce to enormous matrix multiplications, and GPUs — built to compute many simple operations in parallel with very high memory bandwidth — turned out to be the ideal engine for them. Data-center GPUs differ from gaming parts in the ways that matter for AI: tens of gigabytes of high-bandwidth memory, fast chip-to-chip interconnects, and tensor cores specialized for low-precision math.

Nvidia holds the overwhelming share of AI compute, sustained as much by the CUDA software ecosystem as by silicon, with successive generations — A100, H100, and the Blackwell line — each defining an era of training and serving economics. Individual accelerators cost tens of thousands of dollars, and frontier clusters aggregate hundreds of thousands of them.

GPU economics flow directly into token prices: hardware cost, utilization, and energy per token set the floor under what any provider can charge. Rental prices for GPU hours and the cadence of new chip generations are therefore leading indicators for where API pricing heads next.