How to Read AI Benchmarks: 7 Statistical Checks

Quick answer: AI benchmark scores describe performance on one test under one set of conditions. To read a headline, check seven things: the test size and its margin of error, whether a gap between models beats the noise, how many attempts were allowed (pass@1 is one try; pass@k counts a success if any of k tries works), possible contamination, relative versus absolute gains, chart scales, and who ran the test with which tools. On the 30 questions of AIME I and II combined, one question is worth 3.3 points.

Facts checked November 22, 2026.

“New model scores 95% on the hardest math test.” “AI beats experts.” Headlines like these arrive every week, and the numbers behind them are real, but they are easy to misread. A few ideas from high school statistics are enough to tell a breakthrough from noise. Here are seven checks, each with a calculation you can do yourself.

The 7 Checks at a Glance

CheckQuestion to askWarning sign
1. Sample sizeHow many questions?Big claims from a 30-question test
2. Margin of errorIs the gap bigger than the noise?“Model A beats Model B” by 1 or 2 points
3. AttemptsOne try or many?pass@64 or “best of” without saying so
4. ContaminationCould the answers be in the training data?Old, public test questions
5. Relative vs absolutePercent or percentage points?“Errors cut in half” with no starting value
6. ChartsWhere does the y-axis start?Axis starting at 80% to make gaps look huge
7. ConditionsWho ran it, with what tools and compute?Company-run tests with undisclosed settings

1. Sample Size: How Many Questions?

A benchmark score is a sample proportion: the fraction of questions answered correctly. Its standard error is

\[ SE = \sqrt{\frac{p(1-p)}{n}}, \]

where \(p\) is the score and \(n\) the number of questions. A 95% margin of error is about \(1.96 \times SE\). This is a normal approximation that treats questions as independent yes-or-no samples from a larger pool of similar questions. Compare two tests, both with a score of 80%:

Test sizeStandard error95% margin of error
500 questions\(\sqrt{0.8 \times 0.2 / 500} \approx 0.018\)About ±3.5 points
30 questions (AIME I and II combined)\(\sqrt{0.8 \times 0.2 / 30} \approx 0.073\)About ±14 points

Each AIME exam has 15 questions, so many AI reports combine AIME I and II for 30. On 30 questions, a single question is worth \(100 \div 30 \approx 3.3\) points, and a jump from 80% to 87% might be just two lucky answers. For small tests, statisticians prefer the Wilson interval, which never goes above 100%: for 24 out of 30 (80%), it runs from about 62.7% to 90.5%. Repeating the same 30 questions many times does not create more independent questions. In 2024, Anthropic researcher Evan Miller published “Adding Error Bars to Evals,” urging AI labs to report standard errors exactly like this.

2. Is the Difference Bigger Than the Noise?

Model A scores 82% and Model B 80% on the same 500 questions. Treating the two scores as independent samples, the standard error of the difference is

\[ \sqrt{\frac{0.82 \times 0.18}{500} + \frac{0.80 \times 0.20}{500}} \approx 0.025, \]

so the 95% interval for the difference is about \(2 \pm 4.9\) points, from \(-2.9\) to \(+6.9\). Note that you cannot judge this by checking whether the two models’ separate intervals overlap; compute the interval for the difference itself.

Because both models answered the same questions, a paired comparison is more precise. Suppose the results were:

OutcomeQuestions
Both correct380
Only A correct30
Only B correct20
Both wrong70

A scores \(410/500 = 82\%\) and B scores \(400/500 = 80\%\). Only the 50 questions where they disagree carry information about the gap. The paired margin works out to about ±2.8 points, so even this sharper test cannot rule out zero difference. The headline “Model A beats Model B” is not supported.

3. One Attempt or Many?

AI labs report scores in different ways:

  • pass@1: one answer per question, scored once. Closest to how you use a chatbot.
  • pass@k: the question counts as solved if any of \(k\) attempts is right.
  • Majority vote (sometimes “cons@64”): the most common of many answers is scored.

The differences are big. If a model solves a problem 30% of the time per attempt, and attempts were independent, its chance of at least one success in 8 tries would be

\[ 1 – 0.7^{8} \approx 1 – 0.058 = 0.942, \]

about 94%, assuming independent attempts on one problem with a fixed 30% success rate. Real attempts are correlated, so actual gains are smaller, but a pass@8 score and a pass@1 score are not comparable. A real example: in the original January 2025 version of its paper, DeepSeek reported that its R1-Zero model reached 71.0% pass@1 on AIME 2024, rising to 86.7% with majority voting over 64 samples. (Later versions of the paper report different settings, so always cite the version.)

4. Could the Test Be in the Training Data?

If benchmark questions, or close copies, appear on the internet, a model may have seen them in training. This is called contamination. In 2024, Scale AI researchers wrote GSM1k, 1,000 fresh grade-school math problems in the style of the popular GSM8k test, and found accuracy drops of up to 8 percentage points for some models. The gap suggests overfitting to the familiar test, though by itself it does not prove the questions were in the training data. That is why newer benchmarks such as FrontierMath keep most problems private (see FrontierMath explained) and why competition results on brand-new problems carry more weight.

5. Relative vs Absolute Changes

“Our new model cuts math errors by 50%!” could mean errors fell from 40% to 20% or from 2% to 1%. Both are a 50% relative drop; the absolute drops are 20 and 1 percentage points. Likewise, “scores doubled” from 2% to 4% is still a very low score. Always ask for the starting value. Review percent of increase and decrease if this is rusty.

6. Read the Chart, Not Just the Bars

  • Truncated y-axis. If the axis starts at 85%, bars for 88% and 90% look wildly different.
  • Log scales. AI progress charts often use logarithmic axes, where each step means ×10. A straight line on a log chart is exponential growth.
  • Cherry-picked comparisons. Which rival models were left off the chart, and which were tested with older settings?

7. Who Ran the Test, and Under What Conditions?

A score depends on more than the model: whether it could use code or search, how long it could think, how many attempts it got, what prompt was used and how answers were graded. Company-reported scores are often run under the best settings for that company’s model. Independent evaluations, such as those by Epoch AI, are useful cross-checks. Funding, access and versions matter too: the original FrontierMath was funded by OpenAI, which had access to much of the problem set, while Epoch kept a holdout set. And FrontierMath version 2, released June 12, 2026, corrected and removed problems, so scores across versions are not directly comparable.

Putting It Together: A Headline Check

Headline: “Model X jumps to 90% on AIME, beating Model Y’s 83%.”

  1. On 30 questions, that is 27 versus 25 correct: a two-question lead. The Wilson intervals, about 74% to 97% for X and 66% to 93% for Y, overlap heavily.
  2. Is 90% pass@1 or majority vote? Was Model Y scored the same way?
  3. Was it tested on AIME problems released after its training data was collected?
  4. Were tools such as Python allowed, and were the thinking-time budgets equal?

Verdict: an interesting result, but not proof that X is better at math. And AIME accuracy says nothing about whether a model is a good tutor. For what such scores mean in practice, see can AI do math?

Common Misconceptions

“A higher score always means a better model.” Only if the gap is bigger than the margin of error and the conditions match.

“Benchmarks measure real-world ability.” They measure performance on one set of questions under one setup. Saturated benchmarks, where top models all score above 90%, can no longer separate them.

“Statistics is just for scientists.” Reading AI news well now takes the same tools as reading polls: sample size, margin of error and honest comparisons.

Watch: Margin of Error and AI Benchmarks

If you learn better by watching, these videos are worth your time.

Confidence intervals and margin of error | AP Statistics (Khan Academy). A clear lesson on margin of error, the main tool for reading benchmark scores.
Limits of AI benchmarks | Demis Hassabis and Lex Fridman (Lex Clips). The Google DeepMind CEO on why benchmarks saturate and what they miss.

Check Your Understanding

Five quick questions to see how much of this has stuck. Pick an answer and press Submit; you will see a short explanation either way.

Answer each question and press Submit to see your result right away.

1. On a 30-question test, how many points is one question worth?
Found a bug? Let us know!
2. A model scores 80% on 500 questions. What is the 95% margin of error, roughly?
Found a bug? Let us know!
3. Two models score 27/30 and 25/30 on AIME. What is the fairest summary?
Found a bug? Let us know!
4. Errors fall from 2% to 1%. What is the relative drop?
Found a bug? Let us know!
5. If one attempt succeeds 30% of the time, what is the chance of at least one success in 8 independent tries?
Found a bug? Let us know!

Frequently Asked Questions

How do you read AI benchmark results?

Check the number of questions and the margin of error, whether scores are pass@1 or best of many attempts, whether the test could be in the training data, whether gains are relative or absolute, and who ran the test under what conditions.

Why are AI benchmark scores misleading?

Small tests have large margins of error, labs use different scoring methods and settings, test questions may have leaked into training data, and charts often exaggerate small gaps.

What is pass@1 vs pass@k?

pass@1 scores one answer per question. pass@k counts a question as solved if any of k attempts is right, which gives much higher numbers.

What is the margin of error for a benchmark score?

Using the normal approximation, about 1.96 × √(p(1 − p)/n) for a score p on n questions: roughly ±3.5 points for 80% on 500 questions, but about ±14 points on a 30-question test, where a Wilson interval is more accurate.

What is AI benchmark contamination?

When benchmark questions or near copies appear in a model’s training data, so a high score may reflect memory rather than reasoning. A 2024 study with fresh GSM1k problems found drops of up to 8 percentage points for some models, a sign of overfitting.

Keep Exploring

Sources and Further Reading

Related to This Article

What people say about "How to Read AI Benchmarks: 7 Statistical Checks - Effortless Math"?

No one replied yet.

Leave a Reply