AI Scaling Laws Explained: Math, Chinchilla and Limits
Quick answer: AI scaling laws are empirical formulas that estimate how a language model’s prediction error (its loss) falls as you add parameters, training data and compute. The improvements follow power laws: in OpenAI’s 2020 study, loss scaled like N^-0.076 in model size N, so 10 times more parameters cut the measured loss by about 16%. DeepMind’s 2022 Chinchilla study showed that model size and training data should grow together, about 20 tokens per parameter. Scaling still helps, but returns diminish, and lower loss does not guarantee better reasoning.
Facts checked November 22, 2026.
Why do AI companies spend billions building ever-bigger models? Because of a surprising mathematical regularity: as you scale up a language model, its error falls along a smooth, predictable curve. These scaling laws are power laws, the same kind of math you meet with exponents and logarithms, and they have guided AI investment since 2020. Here is the math, with numbers you can check.
What Is a Power Law?
A power law has the form \(y = a \cdot x^{-b}\). Unlike exponential decay, which shrinks by the same factor for every fixed step in \(x\), a power law shrinks by the same factor for every fixed multiplication of \(x\). Take logarithms of both sides:
\[ \log y = \log a – b \log x. \]That is a straight line with slope \(-b\) when you plot \(\log y\) against \(\log x\). This is why scaling-law charts use log-log axes: a straight line on them is the signature of a power law. Need a refresher? See exponential growth and decay to compare the two kinds of curves.
The 2020 Scaling Laws (Kaplan et al.)
In January 2020, Jared Kaplan and colleagues at OpenAI measured the test loss \(L\) (a measure of how badly a model predicts the next word) for many model sizes. They found approximately
\[ L(N) \propto N^{-0.076}, \]where \(N\) is the number of parameters, with similar power laws for dataset size and compute, holding across more than seven orders of magnitude. The exponent is small, so returns are slow but steady:
\[ 10^{-0.076} \approx 0.84. \]Every 10 times more parameters cuts the cross-entropy loss by about 16%, when data and training are not the bottleneck. That is 16% lower loss, not 16% fewer wrong answers. Going from 1 billion to 1 trillion parameters (1,000 times larger) multiplies loss by \(1000^{-0.076} \approx 0.59\), about 41% lower, if the trend holds that far. Within the architectures and ranges they tested, width and depth mattered surprisingly little; scale mattered most.
Chinchilla: The 20 Tokens per Parameter Rule
In March 2022, a DeepMind team led by Jordan Hoffmann asked a sharper question: for a fixed compute budget, how should you split it between model size and training data? After training over 400 models, they found the two should grow together: double the parameters, double the tokens. Their compute-optimal recipe works out to roughly 20 training tokens per parameter.
This is an approximate rule for minimizing training loss on a fixed compute budget, not a universal optimum. They tested it with Chinchilla: 70 billion parameters trained on about 1.4 trillion tokens, using the same compute as their 280-billion-parameter Gopher model. Chinchilla beat Gopher, GPT-3 and other larger models on most of the reported evaluations. Many earlier giant models had been undertrained.
| Model or rule | Parameters | Training tokens | Tokens per parameter |
|---|---|---|---|
| GPT-3 (2020) | 175 billion | About 300 billion | About 1.7 |
| Chinchilla (2022) | 70 billion | About 1.4 trillion | About 20 |
| Llama 3 8B (2024) | 8 billion | About 15 trillion | About 1,875 |
A Worked Example: Sizing a Model
Training compute is roughly \(C \approx 6ND\), where \(N\) is parameters and \(D\) is tokens (see why AI runs on matrix multiplication). With the Chinchilla ratio \(D = 20N\):
\[ C \approx 6N(20N) = 120N^2, \quad \text{so} \quad N \approx \sqrt{\frac{C}{120}}. \]For a budget of \(C = 10^{24}\) operations: \(N \approx \sqrt{8.3 \times 10^{21}} \approx 9.1 \times 10^{10}\), about 91 billion parameters, trained on about \(20 \times 91\) billion \(\approx 1.8\) trillion tokens. Because \(N\) grows like \(\sqrt{C}\), 100 times more compute buys only 10 times more parameters.
Why Companies Now “Overtrain” Small Models
Chinchilla minimizes training cost. But a popular model is run billions of times, so a smaller model is cheaper to serve. Labs now often train smaller models on far more data than 20 tokens per parameter. Meta’s Llama 3 8B, released in 2024, was trained on about 15 trillion tokens, roughly 1,875 tokens per parameter, and Meta reported that its models kept improving well past the compute-optimal amount of data. The scaling laws still apply; the goal being optimized has changed from cheapest training to cheapest total use.
A New Axis: Scaling Thinking Time
In September 2024, OpenAI’s o1 showed a second kind of scaling: accuracy on hard math rose steadily as the model was allowed more test-time compute, longer chains of reasoning before answering. Reasoning models trained with reinforcement learning (see reinforcement learning explained) made this a central way to improve math performance from 2025 on.
Three Kinds of Scaling
| Kind | What grows | Example |
|---|---|---|
| Pretraining scaling | Parameters, data and compute for next-word prediction | Kaplan (2020), Chinchilla (2022) |
| Post-training scaling | Reinforcement learning and fine-tuning compute | Reasoning models trained on checkable math |
| Test-time scaling | Thinking time per question | OpenAI o1 (2024) and later reasoning models |
What the Trend Looks Like in 2026
| Trend | Growth | Source |
|---|---|---|
| Training compute of frontier language models | About 5× per year since 2020 (doubling every 5.2 months) | Epoch AI, updated Nov 5, 2026 |
| Training cost of frontier models | About 3.5× per year | Epoch AI |
| AI chip performance per dollar | About +49% per year since 2023 | Epoch AI |
| Feasible training runs by 2030 | About \(2 \times 10^{29}\) operations, limited first by power, then chips | Epoch AI, Aug 2024 projection |
Check the doubling time: 5× per year means doubling every \(12 \times \frac{\log 2}{\log 5} \approx 5.2\) months. Over the six years from 2020 to 2026, growth of 5× a year would multiply compute by \(5^6 = 15{,}625\), the same order of magnitude as Epoch’s estimate of roughly 10,000 times; the 5× rate is a fitted average, not an exact yearly figure.
Are AI Scaling Laws Slowing in 2026?
Two different questions hide here. Is scaling still being done? Yes: training compute for frontier models kept growing about 5 times per year through 2026, and AI data centers are larger than ever. Is each step paying off as much? By design, no: a power law means every tenfold increase buys a similar percentage cut in loss, so the absolute gains shrink while costs grow. Since 2024, much of the visible progress, especially in math, has come from post-training and test-time scaling rather than from pretraining alone. Diminishing returns are not a hard ceiling, but the cost of the next step keeps rising.
What Scaling Laws Do Not Tell You
- Loss is not ability. Scaling laws predict next-word loss, not whether a model can prove a theorem. Abilities can appear to jump suddenly; a 2023 Stanford paper, “Are Emergent Abilities of Large Language Models a Mirage?”, argued many such jumps come from how tests are scored.
- They are empirical, not laws of nature. They are fitted curves that held over a range and could bend.
- Diminishing returns are built in. Each 16% cut in loss needs 10× more parameters, so costs rise fast.
- Data may run short. High-quality human text is finite; synthetic data and reinforcement learning are partial answers.
Common Misconceptions
“Bigger is always better.” Only with enough data; Chinchilla showed a smaller model trained longer can beat a bigger one.
“Scaling laws prove AI will keep improving forever.” They describe past trends; physical, financial and data limits decide how long they continue.
“Exponential and power-law growth are the same.” No. Exponential growth multiplies by a fixed factor each step; a power law changes by a fixed factor each time the input is multiplied (see is AI growing exponentially?).
Watch: AI Scaling Laws Explained
If you learn better by watching, these videos are worth your time.
Check Your Understanding
Five quick questions to see how much of this has stuck. Pick an answer and press Submit; you will see a short explanation either way.
Answer each question and press Submit to see your result right away.
Frequently Asked Questions
What are AI scaling laws?
Empirical power-law formulas showing that a language model’s loss falls predictably as you increase parameters, training data and compute.
What is the Chinchilla scaling law?
A 2022 DeepMind result that, for a fixed compute budget, model size and training tokens should grow together, roughly 20 tokens per parameter. Their 70-billion-parameter Chinchilla beat the 280-billion-parameter Gopher.
Why do scaling laws use log-log plots?
Because a power law y = a·x^(−b) becomes a straight line with slope −b when both axes are logarithmic.
Are AI scaling laws slowing down?
Scaling continues: training compute grew about 5× per year through 2026. But each tenfold increase buys a similar percentage cut in loss, so absolute gains shrink while costs rise, and power, chips and data may limit growth. Post-training and test-time compute have become major new ways to scale.
What is the difference between Kaplan and Chinchilla scaling laws?
Kaplan et al. (2020) suggested growing model size faster than data. Chinchilla (2022) found that for a fixed compute budget, parameters and training tokens should grow together, about 20 tokens per parameter.
How much does 10 times more compute improve an AI model?
Under the 2020 OpenAI fits, loss falls by roughly 11 to 20% per tenfold increase in compute, parameters or data, so improvements are steady but expensive.
Keep Exploring
- Why AI Runs on Matrix Multiplication (and Why GPUs Matter)
- The Math Behind ChatGPT: Explained With Examples
- Is AI Growing Exponentially? The Math and 2026 Data
Sources and Further Reading
- Kaplan et al., Scaling Laws for Neural Language Models (arXiv 2001.08361)
- Hoffmann et al., Training Compute-Optimal Large Language Models (arXiv 2203.15556)
- Epoch AI: Trends in AI (updated Nov 2026)
- Epoch AI: Can AI scaling continue through 2030? (Aug 20, 2024)
- OpenAI: Learning to reason with LLMs (Sept 12, 2024)
- Schaeffer, Miranda and Koyejo, Are Emergent Abilities of Large Language Models a Mirage? (arXiv 2304.15004)
Related to This Article
More math articles
- Comte, Martineau, and Spencer
- CM to Inches Converter (and Every Length Unit)
- West Virginia WVGSA Grade 6 Math Free Worksheets: Printable Grade 6 Math Practice with Answer Keys
- How to Pass the FTCE Math Test?
- How to Solve Quadratic Inequalities? (+FREE Worksheet!)
- 49 Free SBAC Grade 3 Math Worksheets for South Dakota Students (One Skill Per Page)
- 7th Grade OSTP Math Worksheets: FREE & Printable
- Fitting a Line to Data: Complete Guide with Video and Examples
- The Ultimate 7th Grade MCAP Math Course (+FREE Worksheets)
- Poisson Distribution Calculator (Free)


What people say about "AI Scaling Laws Explained: Math, Chinchilla and Limits - Effortless Math"?
No one replied yet.