AI Scaling Laws Explained: Math, Chinchilla and Limits

Quick answer: AI scaling laws are empirical formulas that estimate how a language model’s prediction error (its loss) falls as you add parameters, training data and compute. The improvements follow power laws: in OpenAI’s 2020 study, loss scaled like N^-0.076 in model size N, so 10 times more parameters cut the measured loss by about 16%. DeepMind’s 2022 Chinchilla study showed that model size and training data should grow together, about 20 tokens per parameter. Scaling still helps, but returns diminish, and lower loss does not guarantee better reasoning.

Facts checked November 22, 2026.

Why do AI companies spend billions building ever-bigger models? Because of a surprising mathematical regularity: as you scale up a language model, its error falls along a smooth, predictable curve. These scaling laws are power laws, the same kind of math you meet with exponents and logarithms, and they have guided AI investment since 2020. Here is the math, with numbers you can check.

What Is a Power Law?

A power law has the form \(y = a \cdot x^{-b}\). Unlike exponential decay, which shrinks by the same factor for every fixed step in \(x\), a power law shrinks by the same factor for every fixed multiplication of \(x\). Take logarithms of both sides:

\[ \log y = \log a – b \log x. \]

That is a straight line with slope \(-b\) when you plot \(\log y\) against \(\log x\). This is why scaling-law charts use log-log axes: a straight line on them is the signature of a power law. Need a refresher? See exponential growth and decay to compare the two kinds of curves.

The 2020 Scaling Laws (Kaplan et al.)

In January 2020, Jared Kaplan and colleagues at OpenAI measured the test loss \(L\) (a measure of how badly a model predicts the next word) for many model sizes. They found approximately

\[ L(N) \propto N^{-0.076}, \]

where \(N\) is the number of parameters, with similar power laws for dataset size and compute, holding across more than seven orders of magnitude. The exponent is small, so returns are slow but steady:

\[ 10^{-0.076} \approx 0.84. \]

Every 10 times more parameters cuts the cross-entropy loss by about 16%, when data and training are not the bottleneck. That is 16% lower loss, not 16% fewer wrong answers. Going from 1 billion to 1 trillion parameters (1,000 times larger) multiplies loss by \(1000^{-0.076} \approx 0.59\), about 41% lower, if the trend holds that far. Within the architectures and ranges they tested, width and depth mattered surprisingly little; scale mattered most.

Chinchilla: The 20 Tokens per Parameter Rule

In March 2022, a DeepMind team led by Jordan Hoffmann asked a sharper question: for a fixed compute budget, how should you split it between model size and training data? After training over 400 models, they found the two should grow together: double the parameters, double the tokens. Their compute-optimal recipe works out to roughly 20 training tokens per parameter.

This is an approximate rule for minimizing training loss on a fixed compute budget, not a universal optimum. They tested it with Chinchilla: 70 billion parameters trained on about 1.4 trillion tokens, using the same compute as their 280-billion-parameter Gopher model. Chinchilla beat Gopher, GPT-3 and other larger models on most of the reported evaluations. Many earlier giant models had been undertrained.

Model or ruleParametersTraining tokensTokens per parameter
GPT-3 (2020)175 billionAbout 300 billionAbout 1.7
Chinchilla (2022)70 billionAbout 1.4 trillionAbout 20
Llama 3 8B (2024)8 billionAbout 15 trillionAbout 1,875

A Worked Example: Sizing a Model

Training compute is roughly \(C \approx 6ND\), where \(N\) is parameters and \(D\) is tokens (see why AI runs on matrix multiplication). With the Chinchilla ratio \(D = 20N\):

\[ C \approx 6N(20N) = 120N^2, \quad \text{so} \quad N \approx \sqrt{\frac{C}{120}}. \]

For a budget of \(C = 10^{24}\) operations: \(N \approx \sqrt{8.3 \times 10^{21}} \approx 9.1 \times 10^{10}\), about 91 billion parameters, trained on about \(20 \times 91\) billion \(\approx 1.8\) trillion tokens. Because \(N\) grows like \(\sqrt{C}\), 100 times more compute buys only 10 times more parameters.

Why Companies Now “Overtrain” Small Models

Chinchilla minimizes training cost. But a popular model is run billions of times, so a smaller model is cheaper to serve. Labs now often train smaller models on far more data than 20 tokens per parameter. Meta’s Llama 3 8B, released in 2024, was trained on about 15 trillion tokens, roughly 1,875 tokens per parameter, and Meta reported that its models kept improving well past the compute-optimal amount of data. The scaling laws still apply; the goal being optimized has changed from cheapest training to cheapest total use.

A New Axis: Scaling Thinking Time

In September 2024, OpenAI’s o1 showed a second kind of scaling: accuracy on hard math rose steadily as the model was allowed more test-time compute, longer chains of reasoning before answering. Reasoning models trained with reinforcement learning (see reinforcement learning explained) made this a central way to improve math performance from 2025 on.

Three Kinds of Scaling

KindWhat growsExample
Pretraining scalingParameters, data and compute for next-word predictionKaplan (2020), Chinchilla (2022)
Post-training scalingReinforcement learning and fine-tuning computeReasoning models trained on checkable math
Test-time scalingThinking time per questionOpenAI o1 (2024) and later reasoning models

What the Trend Looks Like in 2026

TrendGrowthSource
Training compute of frontier language modelsAbout 5× per year since 2020 (doubling every 5.2 months)Epoch AI, updated Nov 5, 2026
Training cost of frontier modelsAbout 3.5× per yearEpoch AI
AI chip performance per dollarAbout +49% per year since 2023Epoch AI
Feasible training runs by 2030About \(2 \times 10^{29}\) operations, limited first by power, then chipsEpoch AI, Aug 2024 projection

Check the doubling time: 5× per year means doubling every \(12 \times \frac{\log 2}{\log 5} \approx 5.2\) months. Over the six years from 2020 to 2026, growth of 5× a year would multiply compute by \(5^6 = 15{,}625\), the same order of magnitude as Epoch’s estimate of roughly 10,000 times; the 5× rate is a fitted average, not an exact yearly figure.

Are AI Scaling Laws Slowing in 2026?

Two different questions hide here. Is scaling still being done? Yes: training compute for frontier models kept growing about 5 times per year through 2026, and AI data centers are larger than ever. Is each step paying off as much? By design, no: a power law means every tenfold increase buys a similar percentage cut in loss, so the absolute gains shrink while costs grow. Since 2024, much of the visible progress, especially in math, has come from post-training and test-time scaling rather than from pretraining alone. Diminishing returns are not a hard ceiling, but the cost of the next step keeps rising.

What Scaling Laws Do Not Tell You

  • Loss is not ability. Scaling laws predict next-word loss, not whether a model can prove a theorem. Abilities can appear to jump suddenly; a 2023 Stanford paper, “Are Emergent Abilities of Large Language Models a Mirage?”, argued many such jumps come from how tests are scored.
  • They are empirical, not laws of nature. They are fitted curves that held over a range and could bend.
  • Diminishing returns are built in. Each 16% cut in loss needs 10× more parameters, so costs rise fast.
  • Data may run short. High-quality human text is finite; synthetic data and reinforcement learning are partial answers.

Common Misconceptions

“Bigger is always better.” Only with enough data; Chinchilla showed a smaller model trained longer can beat a bigger one.

“Scaling laws prove AI will keep improving forever.” They describe past trends; physical, financial and data limits decide how long they continue.

“Exponential and power-law growth are the same.” No. Exponential growth multiplies by a fixed factor each step; a power law changes by a fixed factor each time the input is multiplied (see is AI growing exponentially?).

Watch: AI Scaling Laws Explained

If you learn better by watching, these videos are worth your time.

AI can’t cross this line and we don’t know why. (Welch Labs). A beautifully animated look at the compute-efficient frontier and power laws.
How Scaling Laws Will Determine AI’s Future | YC Decoded (Y Combinator). A short history from Kaplan to Chinchilla to test-time compute.

Check Your Understanding

Five quick questions to see how much of this has stuck. Pick an answer and press Submit; you will see a short explanation either way.

Answer each question and press Submit to see your result right away.

1. If loss falls like N^−0.076, what happens with 10 times more parameters?
Found a bug? Let us know!
2. What does a power law look like on a log-log plot?
Found a bug? Let us know!
3. About how many training tokens per parameter did Chinchilla suggest?
Found a bug? Let us know!
4. With C ≈ 120N², how much more compute do you need for 10 times more parameters?
Found a bug? Let us know!
5. What does test-time scaling increase?
Found a bug? Let us know!

Frequently Asked Questions

What are AI scaling laws?

Empirical power-law formulas showing that a language model’s loss falls predictably as you increase parameters, training data and compute.

What is the Chinchilla scaling law?

A 2022 DeepMind result that, for a fixed compute budget, model size and training tokens should grow together, roughly 20 tokens per parameter. Their 70-billion-parameter Chinchilla beat the 280-billion-parameter Gopher.

Why do scaling laws use log-log plots?

Because a power law y = a·x^(−b) becomes a straight line with slope −b when both axes are logarithmic.

Are AI scaling laws slowing down?

Scaling continues: training compute grew about 5× per year through 2026. But each tenfold increase buys a similar percentage cut in loss, so absolute gains shrink while costs rise, and power, chips and data may limit growth. Post-training and test-time compute have become major new ways to scale.

What is the difference between Kaplan and Chinchilla scaling laws?

Kaplan et al. (2020) suggested growing model size faster than data. Chinchilla (2022) found that for a fixed compute budget, parameters and training tokens should grow together, about 20 tokens per parameter.

How much does 10 times more compute improve an AI model?

Under the 2020 OpenAI fits, loss falls by roughly 11 to 20% per tenfold increase in compute, parameters or data, so improvements are steady but expensive.

Keep Exploring

Sources and Further Reading

Related to This Article

What people say about "AI Scaling Laws Explained: Math, Chinchilla and Limits - Effortless Math"?

No one replied yet.

Leave a Reply