Prompt caching

When many requests share the same opening content — a long system prompt, tool definitions, a reference document — the provider can store the internal state computed for that prefix and skip recomputing it. Cached input tokens are then billed at a steep discount relative to the normal input rate.

Implementations differ: some providers cache automatically on exact prefix matches, while others require explicit cache breakpoints and charge a premium to write the cache before discounted reads apply. Cache entries also expire after minutes to hours of inactivity, so steady traffic benefits far more than sporadic calls.

Caching is one of the largest levers in modern API cost optimization. Agents and chat applications that resend large stable contexts on every turn can see their input bill fall by more than half, but only if prompts are structured so the stable content comes first and stays byte-identical across requests.