HumanEval

Released with OpenAI's Codex paper in 2021, HumanEval gives the model a function signature and docstring and checks generated code against hidden unit tests, reported as pass@k — the probability that at least one of k samples passes. It established executable correctness, rather than text similarity, as the standard for judging code generation.

HumanEval is effectively saturated: even small modern models score in the 90s percent, so it no longer distinguishes anything above the entry level. Contamination is also assumed, since the problems have circulated in public for years.

Its role today is historical reference and sanity check, with real coding comparisons resting on repository-level and agentic benchmarks like SWE-bench and continuously refreshed contest-style suites that resist memorization.