HumanEval Leaderboard
Python function synthesis from docstrings. · Prices checked November 11, 2026
HumanEval asks a model to write a self-contained Python function from its docstring, then runs the function against hidden unit tests. It measures code generation in isolation, no repository, no dependencies, no existing tests to satisfy.
Leaderboard
60 of 205 scoredLeaderboardScores as published by each benchmark. Prices checked November 11, 2026.
o4-mini leads on HumanEval at 97.3%. The median across the 205 models we hold a score for is 76%. The cheapest model on this board is DeepSeek-Coder-V2 at $0.1820/M blended, scoring 90.2%.
| # | Model | Provider | Score | Input /1M | Output /1M | Blended 70/30 | Context |
|---|---|---|---|---|---|---|---|
| 1 | o4-mini | OpenAI | 97.3% | $1.10 | $4.40 | $2.09 | 200,000 |
| 2 | o3-mini | OpenAI | 96.3% | $1.10 | $4.40 | $2.09 | 200,000 |
| 3 | GPT-5 | OpenAI | 96% | $1.25 | $10.00 | $3.88 | 400,000 |
| 4 | Claude Opus 4.6 | Anthropic | 95% | $5.00 | $25.00 | $11.00 | 1,000,000 |
| 5 | GPT-4.1 | OpenAI | 94.5% | $2.00 | $8.00 | $3.80 | 1,000,000 |
| 6 | GPT-4.1 mini | OpenAI | 93.8% | $0.4000 | $1.60 | $0.7600 | 1,000,000 |
| 7 | Claude Sonnet 3.5 | Anthropic | 93.7% | $3.00 | $15.00 | $6.60 | 200,000 |
| 8 | Claude Opus 4.5 | Anthropic | 93% | $5.00 | $25.00 | $11.00 | 200,000 |
| 9 | Qwen2.5-Coder-32B (Groq) | Groq | 92.7% | $0.7900 | $0.7900 | $0.7900 | 128,000 |
| 10 | Qwen2.5-Coder-32B | Qwen | 92.7% | $0.7000 | $0.7000 | $0.7000 | 128,000 |
| 11 | Claude Sonnet 4.6 | Anthropic | 92% | $3.00 | $15.00 | $6.60 | 1,000,000 |
| 12 | Gemini 2.5 Pro | 92% | $1.25 | $10.00 | $3.88 | 1,000,000 | |
| 13 | Mistral Large 2 | Mistral | 92% | $2.00 | $6.00 | $3.20 | 128,000 |
| 14 | Nova Premier | Amazon | 92% | $2.50 | $12.50 | $5.50 | 300,000 |
| 15 | Mistral-Large-2 (NIM) | Nvidia | 92% | $2.00 | $6.00 | $3.20 | 128,000 |
| 16 | Kimi k1.5 | Moonshot AI | 92% | $2.50 | $10.00 | $4.75 | 128,000 |
| 17 | Claude Sonnet 4.5 | Anthropic | 91% | $3.00 | $15.00 | $6.60 | 200,000 |
| 18 | GPT-4o | OpenAI | 90.2% | $2.50 | $10.00 | $4.75 | 128,000 |
| 19 | DeepSeek-Coder-V2 | DeepSeek | 90.2% | $0.1400 | $0.2800 | $0.1820 | 128,000 |
| 20 | DeepSeek-R1 | DeepSeek | 90% | $0.5500 | $2.19 | $1.04 | 64,000 |
| 21 | DeepSeek-R1-0528 | DeepSeek | 90% | $0.5500 | $2.19 | $1.04 | 64,000 |
| 22 | Pixtral Large | Mistral | 90% | $2.00 | $6.00 | $3.20 | 128,000 |
| 23 | DeepSeek-R1 (Groq) | Groq | 90% | $0.7500 | $0.9900 | $0.8220 | 128,000 |
| 24 | DeepSeek-R1 (Together) | Together AI | 90% | $3.00 | $7.00 | $4.20 | 64,000 |
| 25 | DeepSeek-R1 (fw) | Fireworks AI | 90% | $3.00 | $8.00 | $4.50 | 64,000 |
| 26 | DeepSeek-R1 (DI) | DeepInfra | 90% | $0.5500 | $2.19 | $1.04 | 64,000 |
| 27 | DeepSeek-R1 (Cerebras) | Cerebras | 90% | $0.5500 | $0.9900 | $0.6820 | 64,000 |
| 28 | Qwen2.5-Max | Qwen | 90% | $1.60 | $6.40 | $3.04 | 32,000 |
| 29 | o1 | OpenAI | 89% | $15.00 | $60.00 | $28.50 | 200,000 |
| 30 | DeepSeek-V3 | DeepSeek | 89% | $0.3200 | $0.8900 | $0.4910 | 64,000 |
| 31 | DeepSeek-V3-0324 | DeepSeek | 89% | $0.2900 | $1.14 | $0.5450 | 128,000 |
| 32 | Llama 3.1 405B | Meta | 89% | $0.8000 | $0.8000 | $0.8000 | 128,000 |
| 33 | Nova Pro | Amazon | 89% | $0.8000 | $3.20 | $1.52 | 300,000 |
| 34 | Llama 3.1 405B (Groq) | Groq | 89% | $2.99 | $2.99 | $2.99 | 128,000 |
| 35 | Llama 3.1 405B (Together) | Together AI | 89% | $3.50 | $3.50 | $3.50 | 128,000 |
| 36 | Llama 3.1 405B (fw) | Fireworks AI | 89% | $3.00 | $3.00 | $3.00 | 131,000 |
| 37 | Llama 3.1 405B (Rep) | Replicate | 89% | $9.50 | $9.50 | $9.50 | 128,000 |
| 38 | Llama 3.1 405B (DI) | DeepInfra | 89% | $0.8000 | $0.8000 | $0.8000 | 128,000 |
| 39 | Llama-3.1-405B (Cerebras) | Cerebras | 89% | $0.9900 | $0.9900 | $0.9900 | 128,000 |
| 40 | Llama 3.1 405B (Hyp) | Hyperbolic | 89% | $4.00 | $4.00 | $4.00 | 128,000 |
| 41 | Qwen2.5-Coder-14B | Qwen | 89% | $0.3500 | $0.3500 | $0.3500 | 128,000 |
| 42 | Grok 2 | xAI | 88.4% | $2.00 | $10.00 | $4.40 | 131,000 |
| 43 | Grok 2 Vision | xAI | 88.4% | $2.00 | $10.00 | $4.40 | 32,000 |
| 44 | Llama 3.3 70B | Meta | 88.4% | $0.2300 | $0.4000 | $0.2810 | 128,000 |
| 45 | Llama 3.3 70B (Groq) | Groq | 88.4% | $0.5900 | $0.7900 | $0.6500 | 128,000 |
| 46 | Llama 3.3 70B (Together) | Together AI | 88.4% | $0.8800 | $0.8800 | $0.8800 | 128,000 |
| 47 | Llama 3.3 70B (DI) | DeepInfra | 88.4% | $0.2300 | $0.4000 | $0.2810 | 128,000 |
| 48 | Llama-3.3-70B (Cerebras) | Cerebras | 88.4% | $0.5900 | $0.9900 | $0.7100 | 128,000 |
| 49 | Qwen2.5-32B-Instruct | Qwen | 88.4% | $0.7000 | $0.7000 | $0.7000 | 128,000 |
| 50 | GPT-4 Turbo | OpenAI | 88.2% | $10.00 | $30.00 | $16.00 | 128,000 |
| 51 | Claude Haiku 3.5 | Anthropic | 88.1% | $0.8000 | $4.00 | $1.76 | 200,000 |
| 52 | Claude Sonnet 3.7 | Anthropic | 88% | $3.00 | $15.00 | $6.60 | 200,000 |
| 53 | Llama 4 Maverick | Meta | 88% | $0.3500 | $1.00 | $0.5450 | 128,000 |
| 54 | Sonar Reasoning Pro | Perplexity | 88% | $2.00 | $8.00 | $3.80 | 127,000 |
| 55 | o3 | OpenAI | 87.4% | $2.00 | $8.00 | $3.80 | 200,000 |
| 56 | GPT-4o mini | OpenAI | 87.2% | $0.1500 | $0.6000 | $0.2850 | 128,000 |
| 57 | GPT-4.1 nano | OpenAI | 87% | $0.1000 | $0.4000 | $0.1900 | 1,000,000 |
| 58 | Gemini 2.5 Flash | 87% | $0.3000 | $2.50 | $0.9600 | 1,000,000 | |
| 59 | Qwen2.5-72B (Groq) | Groq | 86.6% | $0.7900 | $0.7900 | $0.7900 | 128,000 |
| 60 | Qwen2.5-72B (Together) | Together AI | 86.6% | $1.20 | $1.20 | $1.20 | 32,000 |
34 of the 60 scores shown link to the page they were read on (underlined; hover for publisher, variant and date read). The rest have no recorded source yet and are being worked through.
Most capable models now score very highly here, which is why SWE-bench Verified has largely replaced it as the benchmark that separates coding models.
About these scores
- Python function synthesis from docstrings.
- Scores are as published by the model’s provider or a public leaderboard. Provider-reported figures are self-reported and are not independently re-run by us.
- Where a hosted variant of an open-weights model has no score of its own, it inherits the base model’s, because the weights are the same.
- Prices beside each score are the provider’s published token rates. How the blended rate is calculated.
Showing the top 60 of 205 models we hold a HumanEval score for.
Frequently asked questions
What is the difference between HumanEval and SWE-bench?
HumanEval gives a model an isolated function to write from a docstring. SWE-bench Verified gives it a whole repository and a real issue, and requires a patch that passes the project’s existing tests. HumanEval measures code generation; SWE-bench measures software engineering, and scores on it are far lower across the board.
Other leaderboards
Rankings that weigh price too
Leaderboards are recomputed on every deploy from the catalogue. Prices checked November 11, 2026.