Evals

An eval is a dataset of representative inputs plus a grading method: exact-match checks for classification, executable tests for code, rubric scoring by a judge model for open-ended output. Public benchmarks rank models in general; evals answer the question that actually matters, how well a specific system performs on your workload.

Evals earn their keep at every change boundary: swapping models when a version is deprecated or a cheaper tier looks tempting, editing prompts, or switching providers for the same open model. Without them, regressions surface as user complaints; with them, a candidate change is a measured pass rate.

They are also the instrument of cost optimisation, since downgrade decisions become defensible when a cheaper model demonstrably holds quality on your eval set. Running evals costs tokens itself (judge models included) which is precisely the kind of offline workload batch discounts exist for.