Information Theory Explained: Entropy, Bits and AI

Quick answer: Information theory is the mathematics of measuring, storing and transmitting information, developed by Claude Shannon in 1948. An event with probability p carries −log2(p) bits of information. Shannon entropy, H = −Σ p log2 p, measures average uncertainty: a fair coin has 1 bit, a coin that lands heads 90% of the time about 0.469 bits, and a fair six-sided die about 2.585 bits. Entropy sets the limit for lossless compression, and cross-entropy is the standard training objective for AI language models that predict the next token.

Facts checked November 22, 2026.

In 1948, a 32-year-old engineer at Bell Labs, Claude Shannon, published “A Mathematical Theory of Communication.” It answered a question nobody had made precise: how do you measure information? His answer, built on probability and logarithms, made modern compression, the internet and mobile phones possible, and it sits at the heart of how AI chatbots are trained today.

Information Is Surprise

A message that tells you something you already expected carries little information; a surprising one carries a lot. Shannon defined the information in an outcome with probability \(p\) as

\[ I = \log_2 \frac{1}{p} \text{ bits}. \]
EventProbabilityInformation
Fair coin lands heads1/2\(\log_2 2 = 1\) bit
A fair die shows 61/6\(\log_2 6 \approx 2.58\) bits
Guessing one number from 1 to 1,0241/1,024\(\log_2 1024 = 10\) bits
Something certain10 bits

By convention, a term with \(p = 0\) contributes 0. This is why the game Twenty Questions works: each yes-or-no answer gives at most 1 bit, and \(2^{20} \approx 1{,}000{,}000\), so 20 good questions can pick one item out of about a million.

Entropy: Average Surprise

For a source with outcomes of probabilities \(p_1, p_2, \ldots\), the entropy is the average information:

\[ H = -\sum_i p_i \log_2 p_i. \]

A biased coin that lands heads 90% of the time:

\[ H = -(0.9 \log_2 0.9 + 0.1 \log_2 0.1) \approx 0.9(0.152) + 0.1(3.322) \approx 0.47 \text{ bits}. \]

Less than half a bit per flip, because the result is usually predictable. Entropy is highest when all outcomes are equally likely: 1 bit for a fair coin, \(\log_2 6 \approx 2.58\) bits for a fair die.

Entropy Is the Limit of Compression

Shannon’s source coding theorem says no lossless code can use fewer bits on average than the entropy, in the long run (any particular short message might happen to compress better or worse). Example: four symbols with probabilities \(\tfrac12, \tfrac14, \tfrac18, \tfrac18\).

SymbolProbabilityCodeLength
A1/201
B1/4102
C1/81103
D1/81113

Average length: \(\tfrac12(1) + \tfrac14(2) + \tfrac18(3) + \tfrac18(3) = 1.75\) bits. Entropy: \(\tfrac12(1) + \tfrac14(2) + \tfrac18(3) + \tfrac18(3) = 1.75\) bits. The code is perfect: common symbols get short codes, rare ones long codes, the idea behind Morse code and Huffman coding, which is one stage of the compression pipelines in ZIP, JPEG and MP3. A fixed-length code would need 2 bits per symbol.

How Predictable Is English?

With 26 letters equally likely, each letter would carry \(\log_2 26 \approx 4.7\) bits. Using real letter frequencies (E common, Z rare) lowers it to about 4.1 bits. In 1951, Shannon asked people to guess the next letter of English text and estimated that, using long context, English carries only about 1 bit per letter (an estimate for the texts he studied, not a universal constant). Most of written language is predictable, which is why compression works so well on text, and why predicting the next word is such a powerful training task.

Cross-Entropy: How AI Language Models Are Trained

Entropy measures the uncertainty of the true distribution \(P\). Cross-entropy measures the average bits needed when you encode outcomes from \(P\) using a model’s predicted distribution \(Q\):

\[ H(P, Q) = -\sum_x P(x) \log_2 Q(x). \]

Example: a coin really lands heads 90% of the time, but a model predicts 50/50. Then \(H(P, Q) = -0.9 \log_2 0.5 – 0.1 \log_2 0.5 = 1\) bit, while the true entropy is only about 0.469 bits. The extra 0.531 bits is the KL divergence, the cost of the model’s wrong beliefs. Cross-entropy is always at least the entropy, with equality only when the model is exactly right.

QuantityMeasures
Entropy \(H(P)\)Uncertainty of the real source
Cross-entropy \(H(P, Q)\)Average bits using the model’s predictions
KL divergenceCross-entropy minus entropy: the model’s extra cost
Perplexity\(2^{H(P,Q)}\): the effective number of equally likely choices per token

A language model assigns a probability to every possible next token, and training minimizes the average of \(-\log q(\text{actual next token})\) over enormous amounts of text. If the model gives the true next word probability 0.5, it pays 1 bit; if 0.01, about 6.6 bits. In practice the loss is usually computed with natural logarithms, so it is reported in nats (1 nat \(\approx 1.44\) bits). Lower cross-entropy means better prediction, and by Shannon’s theory better prediction allows better compression. A Google DeepMind paper, “Language Modeling Is Compression” (2023, presented at ICLR 2024), reported that Chinchilla 70B, trained mostly on text, compressed ImageNet image patches to 43.4% and LibriSpeech audio to 16.4% of their raw size, not counting the size of the model itself. See the probability behind AI and AI scaling laws, which are stated in terms of this loss.

Channel Capacity: The Speed Limit of Communication

Shannon also found the maximum error-free data rate of a noisy channel. For bandwidth \(B\) and signal-to-noise ratio \(S/N\),

\[ C = B \log_2\left(1 + \frac{S}{N}\right). \]

A 1 MHz channel with \(S/N = 1{,}000\) (30 decibels) can carry at most \(10^6 \times \log_2 1001 \approx 10\) million bits per second. Wi-Fi, 5G and deep-space links are engineered to approach this limit using error-correcting codes.

Common Misconceptions

“Information means meaning.” Shannon’s information measures uncertainty, not importance; a random string has high entropy but no meaning.

“Compression can always shrink files.” Not below their entropy. Already-compressed or random files barely shrink.

“AI loss is an arbitrary score.” Cross-entropy loss is measured in bits (or nats) per token, a direct information-theory quantity.

Watch: Information Theory and Entropy

If you learn better by watching, these videos are worth your time.

Reinventing Entropy | Information theory, part 1 (3Blue1Brown). A beautiful build-up of entropy from the idea of surprise.
Entropy (for data science) Clearly Explained!!! (StatQuest with Josh Starmer). Step-by-step entropy calculations with simple examples.

Check Your Understanding

Five quick questions to see how much of this has stuck. Pick an answer and press Submit; you will see a short explanation either way.

Answer each question and press Submit to see your result right away.

1. How many bits of information are in an event with probability 1/8?
Found a bug? Let us know!
2. What is the entropy of a coin that lands heads 90% of the time?
Found a bug? Let us know!
3. Codes 0, 10, 110, 111 for probabilities 1/2, 1/4, 1/8, 1/8 give what average length?
Found a bug? Let us know!
4. About how many yes/no questions identify one item among a million?
Found a bug? Let us know!
5. A coin is 90% heads, but a model predicts 50/50. What is the cross-entropy?
Found a bug? Let us know!

Frequently Asked Questions

What is information theory?

The mathematical study of measuring, compressing and transmitting information, founded by Claude Shannon in 1948. It measures information in bits as surprise, log₂(1/p).

What is Shannon entropy?

The average information of a source, H = −Σ p log₂ p. A fair coin has 1 bit of entropy per flip; a fair die about 2.58 bits.

Why is entropy the limit of compression?

Shannon’s source coding theorem shows that no lossless code can use fewer bits on average than the source’s entropy, so entropy is the best possible compression rate.

How is information theory used in AI?

Cross-entropy is the standard training objective for language models: the average number of bits (or nats) needed to encode the actual next token under the model’s predictions. Better prediction means lower cross-entropy and better compression.

What is the difference between entropy and cross-entropy?

Entropy measures the uncertainty of the true distribution. Cross-entropy measures the average bits needed when using a model’s predicted distribution; the difference between them is the KL divergence.

What is channel capacity?

The maximum reliable data rate of a noisy channel, C = B log₂(1 + S/N), from the Shannon–Hartley theorem.

Keep Exploring

Sources and Further Reading

Related to This Article

What people say about "Information Theory Explained: Entropy, Bits and AI - Effortless Math"?

No one replied yet.

Leave a Reply