The Math Behind ChatGPT: Explained With Examples

Quick answer: ChatGPT uses linear algebra, probability and calculus. Its language model turns text into tokens, represents each token as a vector of numbers, processes the vectors through transformer layers that use dot products and matrix multiplication, and assigns a probability to every possible next token. A decoding rule picks a token and the process repeats. Calculus is used earlier, during training, to adjust the model’s learned numbers. A high probability does not guarantee a correct answer.

ChatGPT can seem like magic, but every step it takes is math you can learn in school: vectors, multiplication, exponents, probability and a bit of calculus. The difference is scale: billions of numbers instead of a few. Here is the whole pipeline, one idea at a time, with small examples you can check by hand.

Generating a Reply: The Pipeline

This describes a simplified GPT-style text model.

StageMathematical operationWhat changes
1. TokenizeLookup in a vocabularyText becomes token IDs
2. EmbedVectorsEach token becomes a list of numbers
3. AttendDot products, softmax, weighted averagesEach position gathers information from earlier ones
4. TransformMatrix multiplication, simple functionsVectors are updated layer by layer
5. PredictSoftmax over the vocabularyScores become next-token probabilities
6. DecodeA selection ruleOne token is chosen; repeat
TrainingGenerating a reply
Happens before release, on huge amounts of textHappens every time you chat
Learned numbers change, using calculusLearned numbers stay fixed
Goal: make correct next tokens more likelyGoal: pick a next token, then the next

1. Tokens: Text Becomes Pieces

The model does not read letters or whole words. It splits text into tokens, common chunks such as “math”, “emat” or ” the”. Each token has an ID number. A rough rule for English is about four characters per token. (This is also why chatbots once miscounted letters in words; see why AI gets math wrong.)

2. Embeddings: Words Become Vectors

Each token is turned into a vector, a list of numbers, often thousands long. Words used in similar ways end up with similar vectors. With just two invented numbers, imagine:

\[ \text{cat} = (0.9,\ 0.8), \quad \text{dog} = (0.8,\ 0.9), \quad \text{car} = (-0.7,\ 0.2). \]

“Cat” and “dog” point in similar directions; “car” points elsewhere.

3. Dot Products: Measuring Similarity

To compare two vectors, multiply matching entries and add. This is the dot product:

\[ \text{cat} \cdot \text{dog} = 0.9(0.8) + 0.8(0.9) = 1.44, \qquad \text{cat} \cdot \text{car} = 0.9(-0.7) + 0.8(0.2) = -0.47. \]

A large positive result suggests vectors point in similar directions; a negative one, opposite directions. (Dot products also depend on vector lengths, so they are not pure similarity scores.) A model computes enormous numbers of dot products for every token it writes.

4. Attention: Which Words Matter?

In “The ball didn’t fit in the bag because it was too big,” what does “it” mean? The attention mechanism, introduced in the 2017 paper “Attention Is All You Need,” lets each word score every other word using dot products, then mix in information from the high-scoring ones. In this example, we would want “it” to attend strongly to “ball.” In a GPT-style model, each position can look only at earlier positions.

A tiny attention calculation with invented numbers: a query \(q = (1, 0)\), two keys \(k_1 = (1, 0)\) and \(k_2 = (0, 1)\), and values \(v_1 = (2, 0)\) and \(v_2 = (0, 2)\). The query–key dot products are 1 and 0. Divide by \(\sqrt{2}\) (the vectors have 2 components) and apply softmax: the weights are about 0.670 and 0.330. The output is the weighted average \(0.670 v_1 + 0.330 v_2 \approx (1.34, 0.66)\), a new vector that mixes in information. Real models do this for thousands of positions at once with matrix multiplication, which is why AI chips are built to multiply matrices fast.

5. Softmax: Scores Become Probabilities

At the end, the model gives every possible next token a score. The softmax function turns scores into probabilities that add to 1:

\[ p_i = \frac{e^{s_i}}{\sum_j e^{s_j}}. \]

Example: after “2 + 2 =”, suppose the scores for “4”, “5” and “four” are 2, 1 and 0. Then \(e^2 \approx 7.39\), \(e^1 \approx 2.72\), \(e^0 = 1\), with total about 11.11, so the probabilities are about 0.665, 0.245 and 0.090. This softmax is over the whole vocabulary of possible next tokens, unlike the attention softmax above, which is over positions in the text. A decoding rule then chooses: greedy decoding takes the most likely token, while sampling picks randomly according to the probabilities, often adjusted by settings such as temperature (see how AI predicts the next word). Then the whole process repeats. See probability basics.

6. Training: Learning With Calculus

When training starts from scratch, a model’s numbers (its parameters) begin mostly random. Pretraining shows it huge amounts of text and measures how much probability it gave the token that actually came next. For one token the loss can be \(L = -\ln p\): if \(p = 0.665\), then \(L \approx 0.408\); a perfect \(p = 1\) gives \(L = 0\). Backpropagation uses the chain rule to find how each parameter affects the loss, and gradient descent nudges each one slightly downhill:

\[ w_{\text{new}} = w_{\text{old}} – \eta \frac{\partial L}{\partial w}. \]

Here \(\eta\) is a small step size (see gradient descent explained). Later training stages, such as learning from human feedback, refine the model further. For historical scale, OpenAI’s GPT-3 (2020) had 175 billion parameters; counts for current ChatGPT models are not published.

Why the Math Matters for You

  • Vectors and matrices (linear algebra) are how AI represents everything.
  • Probability explains why answers vary and why confident-sounding text can be wrong.
  • Calculus is how models learn.

Those are exactly the subjects to study if you want to build AI; see math skills for the age of AI.

Common Misconceptions

“ChatGPT looks answers up in a database.” No. It generates text token by token from probabilities learned in training (though some versions can also search the web).

“It understands words like we do.” It represents them as vectors and patterns; whether that counts as understanding is debated.

“The math is too advanced for students.” Each piece is school math: the hard part is scale, not the ideas.

Watch: The Math Behind ChatGPT, Animated

If you learn better by watching, these videos are worth your time.

Large Language Models explained briefly (3Blue1Brown). A short, visual overview of how chatbots work.
Transformers, the tech behind LLMs (3Blue1Brown). Embeddings, vectors and the transformer, animated.
Attention in transformers, step-by-step (3Blue1Brown). How attention uses dot products and matrices.

Check Your Understanding

Five quick questions to see how much of this has stuck. Pick an answer and press Submit; you will see a short explanation either way.

Answer each question and press Submit to see your result right away.

1. What is the dot product of (0.9, 0.8) and (0.8, 0.9)?
Found a bug? Let us know!
2. What does softmax do?
Found a bug? Let us know!
3. Scores 2, 1 and 0 go into softmax. About what probability does the top choice get?
Found a bug? Let us know!
4. Do ChatGPT’s learned numbers change while it answers you?
Found a bug? Let us know!
5. What is a token?
Found a bug? Let us know!

Frequently Asked Questions

What math is used in ChatGPT?

Linear algebra (vectors, dot products and matrix multiplication), probability (softmax over possible next tokens) and calculus (derivatives and gradient descent for training).

How does ChatGPT predict the next word?

It turns the text into vectors, processes them through transformer layers with attention, gives every possible next token a score, converts scores to probabilities with softmax and chooses a likely token, then repeats.

What is attention in a transformer?

A mechanism, introduced in the 2017 paper Attention Is All You Need, in which each token scores the others with dot products and mixes in information from the most relevant ones.

What is softmax?

A function that turns a list of scores into probabilities by exponentiating each score and dividing by the total, so the results are positive and add to 1.

How many parameters does ChatGPT have?

OpenAI has not published parameter counts for recent models. For scale, GPT-3 in 2020 had 175 billion parameters.

Keep Exploring

Sources and Further Reading

Related to This Article

What people say about "The Math Behind ChatGPT: Explained With Examples - Effortless Math"?

No one replied yet.

Leave a Reply