How to Read AI Benchmarks: 7 Statistical Checks
Quick answer: AI benchmark scores describe performance on one test under one set of conditions. To read a headline, check seven things: the test size and its margin of error, whether a gap between models beats the noise, how many attempts were allowed (pass@1 is one try; pass@k counts a success if any of k tries works), possible contamination, relative versus absolute gains, chart scales, and who ran the test with which tools. On the 30 questions of AIME I and II combined, one question is worth 3.3 points.
Facts checked November 22, 2026.
“New model scores 95% on the hardest math test.” “AI beats experts.” Headlines like these arrive every week, and the numbers behind them are real, but they are easy to misread. A few ideas from high school statistics are enough to tell a breakthrough from noise. Here are seven checks, each with a calculation you can do yourself.
The 7 Checks at a Glance
| Check | Question to ask | Warning sign |
|---|---|---|
| 1. Sample size | How many questions? | Big claims from a 30-question test |
| 2. Margin of error | Is the gap bigger than the noise? | “Model A beats Model B” by 1 or 2 points |
| 3. Attempts | One try or many? | pass@64 or “best of” without saying so |
| 4. Contamination | Could the answers be in the training data? | Old, public test questions |
| 5. Relative vs absolute | Percent or percentage points? | “Errors cut in half” with no starting value |
| 6. Charts | Where does the y-axis start? | Axis starting at 80% to make gaps look huge |
| 7. Conditions | Who ran it, with what tools and compute? | Company-run tests with undisclosed settings |
1. Sample Size: How Many Questions?
A benchmark score is a sample proportion: the fraction of questions answered correctly. Its standard error is
\[ SE = \sqrt{\frac{p(1-p)}{n}}, \]where \(p\) is the score and \(n\) the number of questions. A 95% margin of error is about \(1.96 \times SE\). This is a normal approximation that treats questions as independent yes-or-no samples from a larger pool of similar questions. Compare two tests, both with a score of 80%:
| Test size | Standard error | 95% margin of error |
|---|---|---|
| 500 questions | \(\sqrt{0.8 \times 0.2 / 500} \approx 0.018\) | About ±3.5 points |
| 30 questions (AIME I and II combined) | \(\sqrt{0.8 \times 0.2 / 30} \approx 0.073\) | About ±14 points |
Each AIME exam has 15 questions, so many AI reports combine AIME I and II for 30. On 30 questions, a single question is worth \(100 \div 30 \approx 3.3\) points, and a jump from 80% to 87% might be just two lucky answers. For small tests, statisticians prefer the Wilson interval, which never goes above 100%: for 24 out of 30 (80%), it runs from about 62.7% to 90.5%. Repeating the same 30 questions many times does not create more independent questions. In 2024, Anthropic researcher Evan Miller published “Adding Error Bars to Evals,” urging AI labs to report standard errors exactly like this.
2. Is the Difference Bigger Than the Noise?
Model A scores 82% and Model B 80% on the same 500 questions. Treating the two scores as independent samples, the standard error of the difference is
\[ \sqrt{\frac{0.82 \times 0.18}{500} + \frac{0.80 \times 0.20}{500}} \approx 0.025, \]so the 95% interval for the difference is about \(2 \pm 4.9\) points, from \(-2.9\) to \(+6.9\). Note that you cannot judge this by checking whether the two models’ separate intervals overlap; compute the interval for the difference itself.
Because both models answered the same questions, a paired comparison is more precise. Suppose the results were:
| Outcome | Questions |
|---|---|
| Both correct | 380 |
| Only A correct | 30 |
| Only B correct | 20 |
| Both wrong | 70 |
A scores \(410/500 = 82\%\) and B scores \(400/500 = 80\%\). Only the 50 questions where they disagree carry information about the gap. The paired margin works out to about ±2.8 points, so even this sharper test cannot rule out zero difference. The headline “Model A beats Model B” is not supported.
3. One Attempt or Many?
AI labs report scores in different ways:
- pass@1: one answer per question, scored once. Closest to how you use a chatbot.
- pass@k: the question counts as solved if any of \(k\) attempts is right.
- Majority vote (sometimes “cons@64”): the most common of many answers is scored.
The differences are big. If a model solves a problem 30% of the time per attempt, and attempts were independent, its chance of at least one success in 8 tries would be
\[ 1 – 0.7^{8} \approx 1 – 0.058 = 0.942, \]about 94%, assuming independent attempts on one problem with a fixed 30% success rate. Real attempts are correlated, so actual gains are smaller, but a pass@8 score and a pass@1 score are not comparable. A real example: in the original January 2025 version of its paper, DeepSeek reported that its R1-Zero model reached 71.0% pass@1 on AIME 2024, rising to 86.7% with majority voting over 64 samples. (Later versions of the paper report different settings, so always cite the version.)
4. Could the Test Be in the Training Data?
If benchmark questions, or close copies, appear on the internet, a model may have seen them in training. This is called contamination. In 2024, Scale AI researchers wrote GSM1k, 1,000 fresh grade-school math problems in the style of the popular GSM8k test, and found accuracy drops of up to 8 percentage points for some models. The gap suggests overfitting to the familiar test, though by itself it does not prove the questions were in the training data. That is why newer benchmarks such as FrontierMath keep most problems private (see FrontierMath explained) and why competition results on brand-new problems carry more weight.
5. Relative vs Absolute Changes
“Our new model cuts math errors by 50%!” could mean errors fell from 40% to 20% or from 2% to 1%. Both are a 50% relative drop; the absolute drops are 20 and 1 percentage points. Likewise, “scores doubled” from 2% to 4% is still a very low score. Always ask for the starting value. Review percent of increase and decrease if this is rusty.
6. Read the Chart, Not Just the Bars
- Truncated y-axis. If the axis starts at 85%, bars for 88% and 90% look wildly different.
- Log scales. AI progress charts often use logarithmic axes, where each step means ×10. A straight line on a log chart is exponential growth.
- Cherry-picked comparisons. Which rival models were left off the chart, and which were tested with older settings?
7. Who Ran the Test, and Under What Conditions?
A score depends on more than the model: whether it could use code or search, how long it could think, how many attempts it got, what prompt was used and how answers were graded. Company-reported scores are often run under the best settings for that company’s model. Independent evaluations, such as those by Epoch AI, are useful cross-checks. Funding, access and versions matter too: the original FrontierMath was funded by OpenAI, which had access to much of the problem set, while Epoch kept a holdout set. And FrontierMath version 2, released June 12, 2026, corrected and removed problems, so scores across versions are not directly comparable.
Putting It Together: A Headline Check
Headline: “Model X jumps to 90% on AIME, beating Model Y’s 83%.”
- On 30 questions, that is 27 versus 25 correct: a two-question lead. The Wilson intervals, about 74% to 97% for X and 66% to 93% for Y, overlap heavily.
- Is 90% pass@1 or majority vote? Was Model Y scored the same way?
- Was it tested on AIME problems released after its training data was collected?
- Were tools such as Python allowed, and were the thinking-time budgets equal?
Verdict: an interesting result, but not proof that X is better at math. And AIME accuracy says nothing about whether a model is a good tutor. For what such scores mean in practice, see can AI do math?
Common Misconceptions
“A higher score always means a better model.” Only if the gap is bigger than the margin of error and the conditions match.
“Benchmarks measure real-world ability.” They measure performance on one set of questions under one setup. Saturated benchmarks, where top models all score above 90%, can no longer separate them.
“Statistics is just for scientists.” Reading AI news well now takes the same tools as reading polls: sample size, margin of error and honest comparisons.
Watch: Margin of Error and AI Benchmarks
If you learn better by watching, these videos are worth your time.
Check Your Understanding
Five quick questions to see how much of this has stuck. Pick an answer and press Submit; you will see a short explanation either way.
Answer each question and press Submit to see your result right away.
Frequently Asked Questions
How do you read AI benchmark results?
Check the number of questions and the margin of error, whether scores are pass@1 or best of many attempts, whether the test could be in the training data, whether gains are relative or absolute, and who ran the test under what conditions.
Why are AI benchmark scores misleading?
Small tests have large margins of error, labs use different scoring methods and settings, test questions may have leaked into training data, and charts often exaggerate small gaps.
What is pass@1 vs pass@k?
pass@1 scores one answer per question. pass@k counts a question as solved if any of k attempts is right, which gives much higher numbers.
What is the margin of error for a benchmark score?
Using the normal approximation, about 1.96 × √(p(1 − p)/n) for a score p on n questions: roughly ±3.5 points for 80% on 500 questions, but about ±14 points on a 30-question test, where a Wilson interval is more accurate.
What is AI benchmark contamination?
When benchmark questions or near copies appear in a model’s training data, so a high score may reflect memory rather than reasoning. A 2024 study with fresh GSM1k problems found drops of up to 8 percentage points for some models, a sign of overfitting.
Keep Exploring
- FrontierMath Benchmark: Scores, Tier 4 and Limits
- Can AI Do Math? What to Trust in 2026
- How AI Predicts the Next Word: Probability Explained
Sources and Further Reading
- Miller, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv 2411.00640)
- Zhang et al., A Careful Examination of Large Language Model Performance on Grade School Arithmetic (GSM1k, arXiv 2405.00332)
- DeepSeek-AI, DeepSeek-R1 (arXiv 2501.12948)
- Epoch AI: Benchmarking hub
Related to This Article
More math articles
- Chapter 1: Origins of American Government
- ParaPathways Practice Test: 10-Question Mini Set with Answers (2027 Prep)
- The Best Grade 4 Math Book for Iowa Students
- How to Add and Subtract Fractions? (+FREE Worksheet!)
- Keto Macro Calculator: Daily Calories, Fat, Protein and Carbs
- Comprehensive Tests for Convergence and Divergence of Infinite Series
- 4 Times Table
- Top 10 Tips You MUST Know to Retake the ACCUPLACER Math
- Free Grade 3 English Worksheets for Missouri Students
- Free GED Math Worksheets: PDFs with Answers by Topic


What people say about "How to Read AI Benchmarks: 7 Statistical Checks - Effortless Math"?
No one replied yet.