Inference

Training happens once; inference happens on every request, forever. Serving a model means loading its weights onto accelerators and executing a forward pass per generated token, and the aggregate compute spent on inference across the industry now far exceeds what is spent on training.

Inference cost per token is driven by model size, hardware efficiency, and serving optimizations: batching many requests together, caching repeated prompt prefixes, quantizing weights, and speculative decoding, which drafts tokens with a small model and verifies them with the large one. Provider margins live in this gap between serving cost and per-token price.

Everything a buyer pays for on an AI API is inference, so the optimization toolkit — right-sizing the model, caching, batching, trimming prompts — is really a set of strategies for buying fewer or cheaper units of it.