Why AI Runs on Matrix Multiplication (and Why GPUs Matter)
Quick answer: AI uses matrix multiplication to compute many learned weighted sums at once. Each neural network layer multiplies its inputs by a matrix of learned weights, and transformers such as ChatGPT use matrix products for attention and feed-forward layers. Multiplying two n by n matrices the standard way takes n^3 multiplications, so the work grows fast. GPUs speed this up with thousands of parallel cores, special matrix units and fast memory. Matrix products supply most of the arithmetic, though memory and data movement also limit speed.
Facts checked November 22, 2026.
Ask what powers ChatGPT, image generators or self-driving software, and the honest answer is a single school-level operation repeated an astronomical number of times: matrix multiplication. It is why NVIDIA became one of the world’s most valuable companies, why AI uses so much electricity, and why mathematicians and AI systems are racing to multiply matrices with fewer steps.
Matrix Multiplication in One Example
To multiply two matrices, take each row of the first and each column of the second, multiply matching entries and add:
\[ \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} \begin{pmatrix} 5 & 6 \\ 7 & 8 \end{pmatrix} = \begin{pmatrix} 1\cdot5 + 2\cdot7 & 1\cdot6 + 2\cdot8 \\ 3\cdot5 + 4\cdot7 & 3\cdot6 + 4\cdot8 \end{pmatrix} = \begin{pmatrix} 19 & 22 \\ 43 & 50 \end{pmatrix}. \]That took 8 multiplications and 4 additions. In general, multiplying an \(m \times k\) matrix by a \(k \times n\) matrix gives an \(m \times n\) result and takes \(m \cdot k \cdot n\) multiplications. Each entry is a dot product of a row and a column. For a step-by-step lesson, see matrix multiplication.
Where AI Uses Matrix Multiplication
A neural network layer takes an input vector \(x\), multiplies it by a weight matrix \(W\), adds a bias \(b\) and applies a simple function. With small numbers:
\[ W x = \begin{pmatrix} 2 & -1 \\ 0 & 3 \end{pmatrix} \begin{pmatrix} 4 \\ 1 \end{pmatrix} = \begin{pmatrix} 8 – 1 \\ 0 + 3 \end{pmatrix} = \begin{pmatrix} 7 \\ 3 \end{pmatrix}. \]Each output is a weighted sum of the inputs: \(7 = 2 \times 4 + (-1) \times 1\). A full layer computes \(y = \text{activation}(Wx + b)\). Now the key trick: stack several inputs as columns and the same weights handle them all in one matrix product, called a batch:
\[ \begin{pmatrix} 2 & -1 \\ 0 & 3 \end{pmatrix} \begin{pmatrix} 4 & 1 \\ 1 & 2 \end{pmatrix} = \begin{pmatrix} 7 & 0 \\ 3 & 6 \end{pmatrix}. \]Column 1 is the first input’s result; column 2 is a second input, \((1, 2)\), processed with the same reused weights. A real model does the same with matrices that have thousands of rows and columns. In a transformer, the design behind ChatGPT, Gemini and Claude, matrix multiplications appear everywhere:
| Part of the model | What gets multiplied |
|---|---|
| Word embeddings | Each token becomes a long vector of numbers (see word embeddings explained) |
| Attention | \(\text{softmax}\left(QK^{T}/\sqrt{d_k}\right)V\): queries times keys score which words matter to each other, then the scores weight the values |
| Feed-forward layers | Large weight matrices applied to every token |
| Training | The same products again, backward, to compute gradients |
Most of the arithmetic in training and running a large language model is matrix multiplication; activation functions, normalization and moving data between memory and chips make up the rest of the work. For the full picture, see the math of neural networks and linear algebra for AI.
How Much Arithmetic Is That?
Multiplying two \(n \times n\) matrices the standard way needs \(n^3\) multiplications (and almost as many additions, so about \(2n^3\) arithmetic operations):
| Matrix size | Multiplications |
|---|---|
| 2 × 2 | 8 |
| 10 × 10 | 1,000 |
| 1,000 × 1,000 | 1,000,000,000 |
| 10,000 × 10,000 | \(10^{12}\), a trillion |
For a rough estimate, researchers use the rule that training a dense transformer takes about \(C \approx 6ND\) arithmetic operations, where \(N\) is the number of parameters and \(D\) the number of training tokens. With 70 billion parameters and 15 trillion tokens:
\[ 6 \times (7 \times 10^{10}) \times (1.5 \times 10^{13}) \approx 6.3 \times 10^{24} \text{ operations}. \]At an assumed sustained \(10^{15}\) operations per second, that is about 200 years for a single chip. Real training spreads the work across many GPUs: Meta reported about 6.4 million H100 GPU-hours to train the original Llama 3 70B on over 15 trillion tokens.
Why GPUs Are Perfect for AI
Look at the 2 × 2 example again: each of the four entries can be computed without waiting for the others. That independence means the work can be split across thousands of small processors at once. Graphics chips were built for exactly this kind of parallel arithmetic, originally to color millions of pixels. Modern AI chips add tensor cores, circuits that multiply small blocks of a matrix in a single step, often using lower-precision numbers (16-bit or 8-bit) because neural networks tolerate small rounding errors. A CPU has a handful of powerful cores; a GPU has thousands of simpler ones, which is what matrix multiplication rewards.
Speed also depends on memory. During training, and when a chatbot first reads your prompt, big batches let each weight be reused many times, so the chip stays busy. When a chatbot generates its answer one token at a time for a single user, each weight is used only once per step, and the bottleneck becomes moving weights from memory, not arithmetic. That is why AI chips also need very fast memory.
Can We Multiply Matrices With Fewer Steps?
For centuries everyone assumed 8 multiplications were needed for 2 × 2 matrices. In 1969, Volker Strassen found a way with 7, using clever sums and differences. Applied recursively to big matrices, saving one multiplication in eight adds up. The search for even better recipes has become an AI-and-math story:
| Year | Who | Result for 4 × 4 matrices |
|---|---|---|
| Standard method | — | 64 multiplications |
| 1969 | Strassen’s method applied twice | 49 |
| October 2022 | DeepMind’s AlphaTensor (Nature) | 47, but only in arithmetic mod 2 (where \(1 + 1 = 0\), as with even and odd) |
| May 2025 | Google DeepMind’s AlphaEvolve | 48, using complex numbers, valid for ordinary arithmetic |
| June 2025 | Dumas, Pernet and Sedoglavic | 48 with only rational numbers, valid in almost any number system |
The two AI systems worked differently: AlphaTensor used reinforcement learning to search for ways to break the multiplication into fewer pieces, while AlphaEvolve evolves computer programs and keeps the ones an automatic evaluator scores best. Fewer multiplications do not automatically mean faster code on a GPU, where extra additions and memory traffic also cost time. Read more in AlphaEvolve explained and algorithms discovered by AI.
The Race to Lower Omega
Mathematicians measure the best possible speed with an exponent \(\omega\) (omega): the smallest number such that \(n \times n\) matrices can be multiplied in about \(n^{\omega}\) steps. The standard method gives 3. Nobody knows whether \(\omega = 2\), which would be essentially as fast as just reading the numbers.
Reported upper bounds on \(\omega\), checked November 22, 2026:
| Year | Bound | Who | Status |
|---|---|---|---|
| 1969 | 2.807 (\(\log_2 7\)) | Strassen | Classic result |
| 1990 | 2.376 | Coppersmith and Winograd | Classic result |
| 2024 | 2.371339 | Alman, Duan, Vassilevska Williams, Xu, Xu and Zhou | Peer reviewed |
| August 2026 | 2.371177 | Dupont and colleagues, using new optimization methods plus AlphaEvolve | Preprint |
| October 2026 | 2.25 (9/4), over the complex numbers | An internal OpenAI model, in a release of 722 AI-generated math manuscripts on October 6 | Claimed, with a Lean proof; not peer reviewed |
The October 2026 claim would be the biggest drop in decades, from about 2.371 to 2.25, and it arrived with a proof checked by the Lean proof assistant. A Lean check confirms the formal statement, but experts still need to confirm it matches the intended problem, so treat it as a striking claim under review rather than settled history. (A separate October 2026 human preprint by Reza Zadeh refined the older method to 2.37115924.) See formal verification explained.
All of these record algorithms are “galactic”: they only win for matrices far larger than any computer could store, so AI chips do not use them, and none makes today’s AI faster. GPUs mostly use the standard method, tuned to the hardware, because it is simple, numerically stable and fits memory well. The theory still matters: it tells us how far matrix multiplication could ultimately be sped up.
Common Misconceptions
“AI is too complex to be just multiplication.” The behavior is complex; the arithmetic is mostly multiply-and-add, repeated at enormous scale with learned numbers.
“AlphaTensor made ChatGPT faster.” Not directly. Its 47-step result works only in mod 2 arithmetic, and record-setting algorithms are rarely used on GPUs.
“Faster chips are the only way forward.” Better algorithms, lower precision and smarter model designs all cut the cost too.
Watch: Matrix Multiplication and AI
If you learn better by watching, these videos are worth your time.
Check Your Understanding
Five quick questions to see how much of this has stuck. Pick an answer and press Submit; you will see a short explanation either way.
Answer each question and press Submit to see your result right away.
Frequently Asked Questions
Why does AI use matrix multiplication?
Neural networks store what they learn as large matrices of weights. Processing an input means multiplying it by those matrices layer after layer, so almost all the arithmetic in AI is matrix multiplication.
Why do AI models need GPUs?
GPUs contain thousands of cores that compute many matrix entries at the same time, plus tensor cores that multiply small matrix blocks in one step, which makes them far faster than CPUs for AI.
How many multiplications does matrix multiplication take?
The standard method takes n³ multiplications for two n by n matrices: 8 for 2 by 2, and a billion for 1,000 by 1,000. Strassen’s 1969 method uses 7 for 2 by 2.
What is the fastest matrix multiplication algorithm?
In theory, an internal OpenAI model claimed in October 2026 that the exponent ω is at most 2.25 over the complex numbers, with a Lean-checked proof that is not yet peer reviewed; the previous reported bound was about 2.371. These algorithms only help for impossibly large matrices, so GPUs use the standard method.
Did AI find a faster way to multiply matrices?
Yes, for small cases. AlphaTensor (2022) found 47 multiplications for 4 by 4 matrices in mod 2 arithmetic, and AlphaEvolve (2025) found 48 for ordinary arithmetic, beating the 49 from Strassen’s method.
Keep Exploring
- Linear Algebra for AI: Vectors, Matrices and Examples
- AlphaEvolve Explained: Google’s AI Math Discoveries
- AI-Discovered Algorithms: AlphaDev and AlphaTensor
Sources and Further Reading
- Fawzi et al., Discovering faster matrix multiplication algorithms with reinforcement learning (Nature, 2022)
- Google DeepMind: AlphaEvolve (May 14, 2025)
- Dumas, Pernet and Sedoglavic, 4×4 matrices with 48 non-complex multiplications (arXiv 2506.13242)
- Alman et al., More Asymmetry Yields Faster Matrix Multiplication (arXiv 2404.16349)
- Dupont et al., Improving the matrix multiplication exponent with modern optimization and AlphaEvolve (arXiv 2608.16884)
- Zadeh, A faster matrix multiplication algorithm through structured optimization (arXiv 2610.10593)
- OpenAI Developer Community: First look at mathematics manuscripts from an internal frontier model (Oct 6, 2026)
- Zvi Mowshowitz: New Math from OpenAI (Oct 9, 2026)
- Kaplan et al., Scaling Laws for Neural Language Models (arXiv 2001.08361)
Related to This Article
More math articles
- 3rd Grade Math Worksheets: Free Printable PDFs by Topic
- Energy and Simple Machines
- TSI Math FREE Sample Practice Questions
- Single-Skill Grade 5 Math for West Virginia (WVGSA): 49 Free Practice PDFs, No Signup
- How to Solve Rational Exponents and Radicals?
- Colorado CMAS Grade 4 Math Free Worksheets: Printable Grade 4 Math Practice with Full Solutions
- Free GED Math Practice Test and 20 Sample Questions
- What are the 5 Ways to Prove Congruence?
- Nonogram Puzzles
- Unit Circle Calculator (Free Exact Values for Any Angle)


What people say about "Why AI Runs on Matrix Multiplication (and Why GPUs Matter) - Effortless Math"?
No one replied yet.