Information Theory Explained: Entropy, Bits and AI
Quick answer: Information theory is the mathematics of measuring, storing and transmitting information, developed by Claude Shannon in 1948. An event with probability p carries −log2(p) bits of information. Shannon entropy, H = −Σ p log2 p, measures average uncertainty: a fair coin has 1 bit, a coin that lands heads 90% of the time about 0.469 bits, and a fair six-sided die about 2.585 bits. Entropy sets the limit for lossless compression, and cross-entropy is the standard training objective for AI language models that predict the next token.
Facts checked November 22, 2026.
In 1948, a 32-year-old engineer at Bell Labs, Claude Shannon, published “A Mathematical Theory of Communication.” It answered a question nobody had made precise: how do you measure information? His answer, built on probability and logarithms, made modern compression, the internet and mobile phones possible, and it sits at the heart of how AI chatbots are trained today.
Information Is Surprise
A message that tells you something you already expected carries little information; a surprising one carries a lot. Shannon defined the information in an outcome with probability \(p\) as
\[ I = \log_2 \frac{1}{p} \text{ bits}. \]| Event | Probability | Information |
|---|---|---|
| Fair coin lands heads | 1/2 | \(\log_2 2 = 1\) bit |
| A fair die shows 6 | 1/6 | \(\log_2 6 \approx 2.58\) bits |
| Guessing one number from 1 to 1,024 | 1/1,024 | \(\log_2 1024 = 10\) bits |
| Something certain | 1 | 0 bits |
By convention, a term with \(p = 0\) contributes 0. This is why the game Twenty Questions works: each yes-or-no answer gives at most 1 bit, and \(2^{20} \approx 1{,}000{,}000\), so 20 good questions can pick one item out of about a million.
Entropy: Average Surprise
For a source with outcomes of probabilities \(p_1, p_2, \ldots\), the entropy is the average information:
\[ H = -\sum_i p_i \log_2 p_i. \]A biased coin that lands heads 90% of the time:
\[ H = -(0.9 \log_2 0.9 + 0.1 \log_2 0.1) \approx 0.9(0.152) + 0.1(3.322) \approx 0.47 \text{ bits}. \]Less than half a bit per flip, because the result is usually predictable. Entropy is highest when all outcomes are equally likely: 1 bit for a fair coin, \(\log_2 6 \approx 2.58\) bits for a fair die.
Entropy Is the Limit of Compression
Shannon’s source coding theorem says no lossless code can use fewer bits on average than the entropy, in the long run (any particular short message might happen to compress better or worse). Example: four symbols with probabilities \(\tfrac12, \tfrac14, \tfrac18, \tfrac18\).
| Symbol | Probability | Code | Length |
|---|---|---|---|
| A | 1/2 | 0 | 1 |
| B | 1/4 | 10 | 2 |
| C | 1/8 | 110 | 3 |
| D | 1/8 | 111 | 3 |
Average length: \(\tfrac12(1) + \tfrac14(2) + \tfrac18(3) + \tfrac18(3) = 1.75\) bits. Entropy: \(\tfrac12(1) + \tfrac14(2) + \tfrac18(3) + \tfrac18(3) = 1.75\) bits. The code is perfect: common symbols get short codes, rare ones long codes, the idea behind Morse code and Huffman coding, which is one stage of the compression pipelines in ZIP, JPEG and MP3. A fixed-length code would need 2 bits per symbol.
How Predictable Is English?
With 26 letters equally likely, each letter would carry \(\log_2 26 \approx 4.7\) bits. Using real letter frequencies (E common, Z rare) lowers it to about 4.1 bits. In 1951, Shannon asked people to guess the next letter of English text and estimated that, using long context, English carries only about 1 bit per letter (an estimate for the texts he studied, not a universal constant). Most of written language is predictable, which is why compression works so well on text, and why predicting the next word is such a powerful training task.
Cross-Entropy: How AI Language Models Are Trained
Entropy measures the uncertainty of the true distribution \(P\). Cross-entropy measures the average bits needed when you encode outcomes from \(P\) using a model’s predicted distribution \(Q\):
\[ H(P, Q) = -\sum_x P(x) \log_2 Q(x). \]Example: a coin really lands heads 90% of the time, but a model predicts 50/50. Then \(H(P, Q) = -0.9 \log_2 0.5 – 0.1 \log_2 0.5 = 1\) bit, while the true entropy is only about 0.469 bits. The extra 0.531 bits is the KL divergence, the cost of the model’s wrong beliefs. Cross-entropy is always at least the entropy, with equality only when the model is exactly right.
| Quantity | Measures |
|---|---|
| Entropy \(H(P)\) | Uncertainty of the real source |
| Cross-entropy \(H(P, Q)\) | Average bits using the model’s predictions |
| KL divergence | Cross-entropy minus entropy: the model’s extra cost |
| Perplexity | \(2^{H(P,Q)}\): the effective number of equally likely choices per token |
A language model assigns a probability to every possible next token, and training minimizes the average of \(-\log q(\text{actual next token})\) over enormous amounts of text. If the model gives the true next word probability 0.5, it pays 1 bit; if 0.01, about 6.6 bits. In practice the loss is usually computed with natural logarithms, so it is reported in nats (1 nat \(\approx 1.44\) bits). Lower cross-entropy means better prediction, and by Shannon’s theory better prediction allows better compression. A Google DeepMind paper, “Language Modeling Is Compression” (2023, presented at ICLR 2024), reported that Chinchilla 70B, trained mostly on text, compressed ImageNet image patches to 43.4% and LibriSpeech audio to 16.4% of their raw size, not counting the size of the model itself. See the probability behind AI and AI scaling laws, which are stated in terms of this loss.
Channel Capacity: The Speed Limit of Communication
Shannon also found the maximum error-free data rate of a noisy channel. For bandwidth \(B\) and signal-to-noise ratio \(S/N\),
\[ C = B \log_2\left(1 + \frac{S}{N}\right). \]A 1 MHz channel with \(S/N = 1{,}000\) (30 decibels) can carry at most \(10^6 \times \log_2 1001 \approx 10\) million bits per second. Wi-Fi, 5G and deep-space links are engineered to approach this limit using error-correcting codes.
Common Misconceptions
“Information means meaning.” Shannon’s information measures uncertainty, not importance; a random string has high entropy but no meaning.
“Compression can always shrink files.” Not below their entropy. Already-compressed or random files barely shrink.
“AI loss is an arbitrary score.” Cross-entropy loss is measured in bits (or nats) per token, a direct information-theory quantity.
Watch: Information Theory and Entropy
If you learn better by watching, these videos are worth your time.
Check Your Understanding
Five quick questions to see how much of this has stuck. Pick an answer and press Submit; you will see a short explanation either way.
Answer each question and press Submit to see your result right away.
Frequently Asked Questions
What is information theory?
The mathematical study of measuring, compressing and transmitting information, founded by Claude Shannon in 1948. It measures information in bits as surprise, log₂(1/p).
What is Shannon entropy?
The average information of a source, H = −Σ p log₂ p. A fair coin has 1 bit of entropy per flip; a fair die about 2.58 bits.
Why is entropy the limit of compression?
Shannon’s source coding theorem shows that no lossless code can use fewer bits on average than the source’s entropy, so entropy is the best possible compression rate.
How is information theory used in AI?
Cross-entropy is the standard training objective for language models: the average number of bits (or nats) needed to encode the actual next token under the model’s predictions. Better prediction means lower cross-entropy and better compression.
What is the difference between entropy and cross-entropy?
Entropy measures the uncertainty of the true distribution. Cross-entropy measures the average bits needed when using a model’s predicted distribution; the difference between them is the KL divergence.
What is channel capacity?
The maximum reliable data rate of a noisy channel, C = B log₂(1 + S/N), from the Shannon–Hartley theorem.
Keep Exploring
- The Math Behind ChatGPT: Explained With Examples
- How AI Predicts the Next Word: Probability Explained
- Why Does AI Hallucinate? Math, Examples and Fixes
Sources and Further Reading
Related to This Article
More math articles
- The Ultimate TASC Math Formula Cheat Sheet
- qSOFA Calculator
- SSAT Upper Level Math Practice Test Questions
- The Best Algebra 1 Book for Washington Students
- New York Algebra I Worksheets and Regents Review
- Government, Economic Systems, Globalization, and Trade
- South Carolina SC READY Grade 3 Math Worksheets — 49 Free Printable Single-Skill PDFs
- Ace the Math Subtests of the ASVAB Test
- How to Graph Polar Equations
- Free Grade 6 English Worksheets for South Dakota Students


What people say about "Information Theory Explained: Entropy, Bits and AI - Effortless Math"?
No one replied yet.