Latency

Two numbers characterize responsiveness: time to first token, dominated by prompt processing and queueing, and inter-token time, which sets how fast text streams once it starts. A long prompt raises TTFT; a slow decode rate stretches the whole response.

Latency scales with model size and prompt length, spikes under provider load, and balloons when reasoning modes think for thousands of hidden tokens before the first visible one. Providers increasingly sell explicit speed tiers — latency-optimized endpoints at premium prices and slower discounted ones — making the latency-cost tradeoff a line item rather than an accident.

The right target depends on the product: interactive chat lives and dies by TTFT, agent pipelines care about end-to-end task time across many calls, and batch workloads should ignore latency entirely and take the discount.