Why AI Runs on Matrix Multiplication (and Why GPUs Matter)

Quick answer: AI uses matrix multiplication to compute many learned weighted sums at once. Each neural network layer multiplies its inputs by a matrix of learned weights, and transformers such as ChatGPT use matrix products for attention and feed-forward layers. Multiplying two n by n matrices the standard way takes n^3 multiplications, so the work grows fast. GPUs speed this up with thousands of parallel cores, special matrix units and fast memory. Matrix products supply most of the arithmetic, though memory and data movement also limit speed.

Facts checked November 22, 2026.

Ask what powers ChatGPT, image generators or self-driving software, and the honest answer is a single school-level operation repeated an astronomical number of times: matrix multiplication. It is why NVIDIA became one of the world’s most valuable companies, why AI uses so much electricity, and why mathematicians and AI systems are racing to multiply matrices with fewer steps.

Matrix Multiplication in One Example

To multiply two matrices, take each row of the first and each column of the second, multiply matching entries and add:

\[ \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} \begin{pmatrix} 5 & 6 \\ 7 & 8 \end{pmatrix} = \begin{pmatrix} 1\cdot5 + 2\cdot7 & 1\cdot6 + 2\cdot8 \\ 3\cdot5 + 4\cdot7 & 3\cdot6 + 4\cdot8 \end{pmatrix} = \begin{pmatrix} 19 & 22 \\ 43 & 50 \end{pmatrix}. \]

That took 8 multiplications and 4 additions. In general, multiplying an \(m \times k\) matrix by a \(k \times n\) matrix gives an \(m \times n\) result and takes \(m \cdot k \cdot n\) multiplications. Each entry is a dot product of a row and a column. For a step-by-step lesson, see matrix multiplication.

Where AI Uses Matrix Multiplication

A neural network layer takes an input vector \(x\), multiplies it by a weight matrix \(W\), adds a bias \(b\) and applies a simple function. With small numbers:

\[ W x = \begin{pmatrix} 2 & -1 \\ 0 & 3 \end{pmatrix} \begin{pmatrix} 4 \\ 1 \end{pmatrix} = \begin{pmatrix} 8 – 1 \\ 0 + 3 \end{pmatrix} = \begin{pmatrix} 7 \\ 3 \end{pmatrix}. \]

Each output is a weighted sum of the inputs: \(7 = 2 \times 4 + (-1) \times 1\). A full layer computes \(y = \text{activation}(Wx + b)\). Now the key trick: stack several inputs as columns and the same weights handle them all in one matrix product, called a batch:

\[ \begin{pmatrix} 2 & -1 \\ 0 & 3 \end{pmatrix} \begin{pmatrix} 4 & 1 \\ 1 & 2 \end{pmatrix} = \begin{pmatrix} 7 & 0 \\ 3 & 6 \end{pmatrix}. \]

Column 1 is the first input’s result; column 2 is a second input, \((1, 2)\), processed with the same reused weights. A real model does the same with matrices that have thousands of rows and columns. In a transformer, the design behind ChatGPT, Gemini and Claude, matrix multiplications appear everywhere:

Part of the modelWhat gets multiplied
Word embeddingsEach token becomes a long vector of numbers (see word embeddings explained)
Attention\(\text{softmax}\left(QK^{T}/\sqrt{d_k}\right)V\): queries times keys score which words matter to each other, then the scores weight the values
Feed-forward layersLarge weight matrices applied to every token
TrainingThe same products again, backward, to compute gradients

Most of the arithmetic in training and running a large language model is matrix multiplication; activation functions, normalization and moving data between memory and chips make up the rest of the work. For the full picture, see the math of neural networks and linear algebra for AI.

How Much Arithmetic Is That?

Multiplying two \(n \times n\) matrices the standard way needs \(n^3\) multiplications (and almost as many additions, so about \(2n^3\) arithmetic operations):

Matrix sizeMultiplications
2 × 28
10 × 101,000
1,000 × 1,0001,000,000,000
10,000 × 10,000\(10^{12}\), a trillion

For a rough estimate, researchers use the rule that training a dense transformer takes about \(C \approx 6ND\) arithmetic operations, where \(N\) is the number of parameters and \(D\) the number of training tokens. With 70 billion parameters and 15 trillion tokens:

\[ 6 \times (7 \times 10^{10}) \times (1.5 \times 10^{13}) \approx 6.3 \times 10^{24} \text{ operations}. \]

At an assumed sustained \(10^{15}\) operations per second, that is about 200 years for a single chip. Real training spreads the work across many GPUs: Meta reported about 6.4 million H100 GPU-hours to train the original Llama 3 70B on over 15 trillion tokens.

Why GPUs Are Perfect for AI

Look at the 2 × 2 example again: each of the four entries can be computed without waiting for the others. That independence means the work can be split across thousands of small processors at once. Graphics chips were built for exactly this kind of parallel arithmetic, originally to color millions of pixels. Modern AI chips add tensor cores, circuits that multiply small blocks of a matrix in a single step, often using lower-precision numbers (16-bit or 8-bit) because neural networks tolerate small rounding errors. A CPU has a handful of powerful cores; a GPU has thousands of simpler ones, which is what matrix multiplication rewards.

Speed also depends on memory. During training, and when a chatbot first reads your prompt, big batches let each weight be reused many times, so the chip stays busy. When a chatbot generates its answer one token at a time for a single user, each weight is used only once per step, and the bottleneck becomes moving weights from memory, not arithmetic. That is why AI chips also need very fast memory.

Can We Multiply Matrices With Fewer Steps?

For centuries everyone assumed 8 multiplications were needed for 2 × 2 matrices. In 1969, Volker Strassen found a way with 7, using clever sums and differences. Applied recursively to big matrices, saving one multiplication in eight adds up. The search for even better recipes has become an AI-and-math story:

YearWhoResult for 4 × 4 matrices
Standard method—64 multiplications
1969Strassen’s method applied twice49
October 2022DeepMind’s AlphaTensor (Nature)47, but only in arithmetic mod 2 (where \(1 + 1 = 0\), as with even and odd)
May 2025Google DeepMind’s AlphaEvolve48, using complex numbers, valid for ordinary arithmetic
June 2025Dumas, Pernet and Sedoglavic48 with only rational numbers, valid in almost any number system

The two AI systems worked differently: AlphaTensor used reinforcement learning to search for ways to break the multiplication into fewer pieces, while AlphaEvolve evolves computer programs and keeps the ones an automatic evaluator scores best. Fewer multiplications do not automatically mean faster code on a GPU, where extra additions and memory traffic also cost time. Read more in AlphaEvolve explained and algorithms discovered by AI.

The Race to Lower Omega

Mathematicians measure the best possible speed with an exponent \(\omega\) (omega): the smallest number such that \(n \times n\) matrices can be multiplied in about \(n^{\omega}\) steps. The standard method gives 3. Nobody knows whether \(\omega = 2\), which would be essentially as fast as just reading the numbers.

Reported upper bounds on \(\omega\), checked November 22, 2026:

YearBoundWhoStatus
19692.807 (\(\log_2 7\))StrassenClassic result
19902.376Coppersmith and WinogradClassic result
20242.371339Alman, Duan, Vassilevska Williams, Xu, Xu and ZhouPeer reviewed
August 20262.371177Dupont and colleagues, using new optimization methods plus AlphaEvolvePreprint
October 20262.25 (9/4), over the complex numbersAn internal OpenAI model, in a release of 722 AI-generated math manuscripts on October 6Claimed, with a Lean proof; not peer reviewed

The October 2026 claim would be the biggest drop in decades, from about 2.371 to 2.25, and it arrived with a proof checked by the Lean proof assistant. A Lean check confirms the formal statement, but experts still need to confirm it matches the intended problem, so treat it as a striking claim under review rather than settled history. (A separate October 2026 human preprint by Reza Zadeh refined the older method to 2.37115924.) See formal verification explained.

All of these record algorithms are “galactic”: they only win for matrices far larger than any computer could store, so AI chips do not use them, and none makes today’s AI faster. GPUs mostly use the standard method, tuned to the hardware, because it is simple, numerically stable and fits memory well. The theory still matters: it tells us how far matrix multiplication could ultimately be sped up.

Common Misconceptions

“AI is too complex to be just multiplication.” The behavior is complex; the arithmetic is mostly multiply-and-add, repeated at enormous scale with learned numbers.

“AlphaTensor made ChatGPT faster.” Not directly. Its 47-step result works only in mod 2 arithmetic, and record-setting algorithms are rarely used on GPUs.

“Faster chips are the only way forward.” Better algorithms, lower precision and smarter model designs all cut the cost too.

Watch: Matrix Multiplication and AI

If you learn better by watching, these videos are worth your time.

How AI Discovered a Faster Matrix Multiplication Algorithm (Quanta Magazine). The AlphaTensor story, with Strassen’s trick animated.
The fastest matrix multiplication algorithm (Dr. Trefor Bazett). A math professor walks through Strassen’s algorithm and why it is faster.

Check Your Understanding

Five quick questions to see how much of this has stuck. Pick an answer and press Submit; you will see a short explanation either way.

Answer each question and press Submit to see your result right away.

1. In the example, what is the top-left entry: 1 × 5 + 2 × 7?
Found a bug? Let us know!
2. How many multiplications does the standard method need for two 10 × 10 matrices?
Found a bug? Let us know!
3. How many multiplications did Strassen need for 2 × 2 matrices?
Found a bug? Let us know!
4. Why are GPUs good at matrix multiplication?
Found a bug? Let us know!
5. What did AlphaTensor find in 2022?
Found a bug? Let us know!

Frequently Asked Questions

Why does AI use matrix multiplication?

Neural networks store what they learn as large matrices of weights. Processing an input means multiplying it by those matrices layer after layer, so almost all the arithmetic in AI is matrix multiplication.

Why do AI models need GPUs?

GPUs contain thousands of cores that compute many matrix entries at the same time, plus tensor cores that multiply small matrix blocks in one step, which makes them far faster than CPUs for AI.

How many multiplications does matrix multiplication take?

The standard method takes n³ multiplications for two n by n matrices: 8 for 2 by 2, and a billion for 1,000 by 1,000. Strassen’s 1969 method uses 7 for 2 by 2.

What is the fastest matrix multiplication algorithm?

In theory, an internal OpenAI model claimed in October 2026 that the exponent ω is at most 2.25 over the complex numbers, with a Lean-checked proof that is not yet peer reviewed; the previous reported bound was about 2.371. These algorithms only help for impossibly large matrices, so GPUs use the standard method.

Did AI find a faster way to multiply matrices?

Yes, for small cases. AlphaTensor (2022) found 47 multiplications for 4 by 4 matrices in mod 2 arithmetic, and AlphaEvolve (2025) found 48 for ordinary arithmetic, beating the 49 from Strassen’s method.

Keep Exploring

Sources and Further Reading

Related to This Article

What people say about "Why AI Runs on Matrix Multiplication (and Why GPUs Matter) - Effortless Math"?

No one replied yet.

Leave a Reply