How AI Predicts the Next Word: Probability Explained
Quick answer: A typical AI chatbot generates text by predicting the next token, a small unit such as a word, word part or punctuation mark. Its language model produces a score for every possible next token, softmax converts the scores into probabilities, and a decoding rule selects one; then the process repeats. A setting called temperature reshapes those probabilities when the model samples. Because the probabilities describe likely continuations, not truth, fluent answers can still contain false claims, called hallucinations.
Every sentence a chatbot writes is built one small piece at a time, and each piece is a probability decision. Once you see the probability behind it, many of AI’s strengths and quirks make sense, from creative writing to confident mistakes.
Next-Token Prediction in One Line
A language model estimates
\[ P(\text{next token} \mid \text{all the text so far}), \]a conditional probability: the chance of each possible next token, given the context. It picks a token, adds it to the text, and repeats. See how to calculate probability for the basics.
A Simple Model You Can Build by Hand
In his 1948 paper, Claude Shannon explored word-transition probabilities and showed that even simple counting produces text that looks a little like English. Here is our own invented counting example: suppose in some text the word “math” is followed by “is” 6 times, “class” 3 times and “test” 1 time. Then a simple model estimates
\[ P(\text{is} \mid \text{math}) = \tfrac{6}{10} = 0.6, \quad P(\text{class} \mid \text{math}) = 0.3, \quad P(\text{test} \mid \text{math}) = 0.1. \]Modern models replace counting with a neural network that looks at the whole context, not just the last word, but the output is still a probability for each next token.
From Scores to Probabilities: Softmax
The network outputs a score for each token. Softmax turns scores \(s_i\) into probabilities:
\[ p_i = \frac{e^{s_i}}{\sum_j e^{s_j}}. \]With scores 2, 1 and 0 for three candidate tokens, the probabilities are about 0.665, 0.245 and 0.090. See the math behind ChatGPT for the rest of the pipeline.
Temperature: Predictable or Creative?
Before softmax, scores are divided by a number called the temperature \(T\). Using the same scores 2, 1, 0:
| Temperature | Probabilities | Effect |
|---|---|---|
| \(T = 0.5\) | 0.867, 0.117, 0.016 | Focused, predictable |
| \(T = 1\) | 0.665, 0.245, 0.090 | Balanced |
| \(T = 2\) | 0.506, 0.307, 0.186 | Varied, more surprising |
(Rounded probabilities may not add to exactly 1.) Temperature changes how the model samples, not what it learned. Scaling by a positive temperature keeps the ranking of tokens the same, so it does not change greedy selection, and a lower temperature does not guarantee a true answer. Lower settings tend to suit factual tasks; higher ones suit brainstorming. Sampling is also why the same question can get different answers.
Sampling: Choosing the Token
- Greedy: always pick the most likely token.
- Sampling: pick randomly according to the probabilities.
- Top-k sampling: sample only from the k most likely tokens.
- Top-p (nucleus) sampling: sample from the smallest set of most likely tokens whose probabilities reach a threshold such as 0.9. With our probabilities 0.665, 0.245 and 0.090, the first two total 0.910, so they are kept and renormalized to about 0.731 and 0.269; the third is dropped.
- Stopping: generation ends when the model produces a special end token or reaches a length limit.
Why AI “Hallucinates”
Token probabilities describe predicted continuations, not whether a claim is true. When evidence is missing or learned patterns are unreliable, a plausible-sounding citation or number can be generated even if it is false. A 2025 OpenAI analysis also argued that training and evaluation can reward guessing over admitting uncertainty. In math, a likely-looking digit can be the wrong digit, which is one reason chatbots once made arithmetic slips (see why AI gets math wrong). Modern systems reduce this with reasoning steps, tools and retrieval of sources, but the underlying engine is still probabilistic, so check important answers.
Probability in Training
During pretraining, the model is scored by how much probability it gave the token that actually came next in the training text. For that observed token the loss is \(L = -\ln p\): for example, \(p = 0.665\) gives \(L \approx 0.408\); \(p = 1\) gives 0, and the loss grows large as \(p\) approaches 0. Gradient descent lowers this loss across trillions of examples (see gradient descent explained).
Common Misconceptions
“AI picks words at random.” No. It samples from carefully computed probabilities, usually favoring likely tokens.
“High probability means true.” No. Likely-sounding text can still be false.
“Only words are predicted.” Tokens can be word pieces, numbers, code symbols or even parts of images in other models.
Watch: How AI Predicts the Next Word
If you learn better by watching, these videos are worth your time.
Check Your Understanding
Five quick questions to see how much of this has stuck. Pick an answer and press Submit; you will see a short explanation either way.
Answer each question and press Submit to see your result right away.
Frequently Asked Questions
How does AI predict the next word?
It computes a probability for every possible next token given the text so far, using a neural network and the softmax function, then a decoding rule picks one, adds it to the text and repeats.
Does a lower temperature make AI more accurate?
Not necessarily. It makes choices more predictable, but it does not change what the model learned and cannot guarantee a true answer.
What is temperature in AI?
A setting that divides the model’s scores before softmax. Low temperature makes outputs focused and predictable; high temperature makes them more varied and creative.
Why do chatbots give different answers to the same question?
Because they often sample from probabilities rather than always picking the single most likely token, and settings like temperature add variety.
Why does AI hallucinate?
It is trained to produce likely text, not to check facts, so plausible but false details can be generated. Tools, retrieval and reasoning reduce but do not remove this.
What probability is used to train language models?
The model is scored on the probability it gave the correct next token, using the loss −ln(p). Training lowers this loss with gradient descent.
Keep Exploring
- The Math Behind ChatGPT: Explained With Examples
- Why Is AI Bad at Math? ChatGPT Mistakes Explained
- Gradient Descent Explained Simply: How AI Learns
Sources and Further Reading
- Shannon, A Mathematical Theory of Communication (Bell System Technical Journal, 1948)
- Holtzman et al., The Curious Case of Neural Text Degeneration (top-p sampling; ICLR 2020, arXiv 1904.09751)
- Kalai et al., Why Language Models Hallucinate (OpenAI, 2025, arXiv 2509.04664)
- Vaswani et al., Attention Is All You Need (arXiv 1706.03762)
Related to This Article
More math articles
- Best Tips for Math Success
- The Best Grade 8 Math Book for Connecticut Students
- Free Grade 3 Math Worksheet Library for Wyoming (WY-TOPP): 49 Printable Skill Pages
- Best Mobile Laptop Stands for Presentation in Schools
- 6 Keys to Studying Effectively for Your Math Class
- Region
- Free Grade 4 English Worksheets for Texas Students
- How to Break Down Complex Math Problems in Your Homework
- Summer Math Workbooks for Grades 3–8: How to Stop Learning Loss Before It Starts
- Expert Advice on Making Math Fun


What people say about "How AI Predicts the Next Word: Probability Explained - Effortless Math"?
No one replied yet.