Throughput

Throughput is quoted at two levels: output tokens per second for a single request — the speed a user perceives while text streams — and aggregate tokens per minute a deployment can sustain across concurrent requests, which is what rate limits cap.

Providers tune a fundamental tradeoff: batching many requests per accelerator maximizes hardware utilization and aggregate throughput but adds queueing that hurts per-request latency. Specialized inference hardware competes largely on breaking this tradeoff, advertising hundreds or thousands of output tokens per second on open models.

For high-volume pipelines and multi-step agents, sustained throughput often matters more than the per-token price, since a cheap model served slowly can make a workload take longer than it is worth. Measured tokens-per-second-per-dollar is the honest comparison metric.