Benchmark

Benchmarks give the industry a shared yardstick: every new model release arrives with a table of scores on tests like MMLU, GPQA, and SWE-bench, and buyers use those tables as a first filter when comparing models.

The yardstick bends, though. Test questions leak into training data (contamination), top models cluster near the ceiling on older tests (saturation), and labs tune releases toward headline benchmarks. Scores are best read as rough capability tiers, not precise rankings, and provider-reported numbers deserve more skepticism than independently reproduced ones.

For purchasing decisions, the productive framing is capability per dollar: two models a few points apart on a benchmark can differ five-fold in per-token price, and nothing replaces evaluating candidates on your own task before committing volume.