How AI Image Generators Work: Diffusion Math Explained

Quick answer: Many AI image generators, including Stable Diffusion, are built on diffusion models. A neural network is trained on images mixed with different amounts of random Gaussian noise and learns to predict that noise. To generate, the system starts from random noise and refines it step by step, guided by the text prompt, until a new image appears. Stable Diffusion speeds this up by working on a compressed version of the image called a latent. The core equation, x_t = √(ᾱ_t)·x_0 + √(1 − ᾱ_t)·ε, mixes a clean image x_0 with noise ε at noise level t. Some newer systems use related methods such as flow matching.

Facts checked November 22, 2026.

Type “a cat reading a math book” and seconds later you get a picture that never existed. The engine behind many AI image and video generators is a diffusion model or a close relative, and its core idea is surprisingly simple: learn to remove noise. Here is the math, step by step, with a live demo where you can watch an image dissolve into static and come back.

How Does AI Turn Noise Into an Image?

Destroying an image is easy: keep adding random noise until it is pure static. Diffusion models learn to run that process backward. During training, a neural network sees millions of images at every level of noise and learns to guess the noise that was added. To create a new image, it starts from pure static and removes a little of its predicted noise at a time, often over 20 to 1,000 steps, until a clean picture remains. The text prompt steers which picture emerges.

The Math of Adding Noise (the Forward Process)

An image is just a list of numbers: a 512 × 512 color image has \(512 \times 512 \times 3 = 786{,}432\) pixel values. Call the clean image \(x_0\). At each step \(t\), we mix in a little Gaussian (bell-curve) noise. A neat property of Gaussians lets us jump straight to any step:

\[ x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 – \bar\alpha_t}\, \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0, I). \]
  • \(\varepsilon\) is fresh random noise with mean 0 and variance 1 in every pixel.
  • \(\bar\alpha_t\) starts near 1 (almost all image) and shrinks toward 0 (almost all noise) as \(t\) grows.
  • The squared weights add to 1: \(\bar\alpha_t + (1 – \bar\alpha_t) = 1\). If the image values and the noise are independent with variance 1, the mix also has variance 1, so the overall spread of values stays controlled (though not the exact brightness of every pixel).

This formula is the result of applying a one-step rule many times: each step does \(x_t = \sqrt{1-\beta_t}\, x_{t-1} + \sqrt{\beta_t}\, \varepsilon_t\), slightly shrinking the image and adding a little noise.

Worked Example: One Noisy Pixel

Suppose a pixel has value \(x_0 = 0.8\), the noise level gives \(\bar\alpha_t = 0.64\), and the random noise sample is \(\varepsilon = -0.5\). Then

\[ x_t = \sqrt{0.64}\,(0.8) + \sqrt{0.36}\,(-0.5) = 0.8(0.8) + 0.6(-0.5) = 0.64 – 0.30 = 0.34. \]

If you know the noise, you can solve backward: \(x_0 = \dfrac{x_t – \sqrt{1-\bar\alpha_t}\,\varepsilon}{\sqrt{\bar\alpha_t}} = \dfrac{0.34 – 0.6(-0.5)}{0.8} = \dfrac{0.64}{0.8} = 0.8.\) A real generator never knows the true noise; it has to predict it. Try it yourself with \(\varepsilon = 0.5\): you should get \(x_t = 0.94\).

In the classic 2020 setup (DDPM), there are \(T = 1000\) steps, and each step adds noise with variance \(\beta_t\) rising from 0.0001 to 0.02; then \(\bar\alpha_t = \prod_{s=1}^{t} (1 – \beta_s)\).

Try It: Add Noise, Then Remove It

Move the slider to choose a step \(t\). The demo uses the exact DDPM schedule above on a small test image. The third panel is an oracle reconstruction: it subtracts the true noise, which only works because the demo knows it. The “imperfect predictor” option simulates a model whose noise guess is off by 20%. This shows the forward math, not a trained generator.

Clean image x₀
Noisy image xₜ
Recovered estimate

Notice two things. By step 600, \(\bar\alpha_t \approx 0.026\): only about 2.6% of the variance comes from the image, so it is very hard to see, although some signal remains mathematically. And with the imperfect predictor switched on, trying to undo heavy noise in one jump fails badly, because a small error in the noise estimate is divided by a tiny \(\sqrt{\bar\alpha_t}\). That is why generation takes many small steps.

What the Neural Network Learns

The network \(\varepsilon_\theta(x_t, t)\) is shown a noisy image and the step number and asked to predict the noise \(\varepsilon\). Training minimizes the average squared error:

\[ L = \mathbb{E}\left[\, \lVert \varepsilon – \varepsilon_\theta(x_t, t) \rVert^2 \,\right]. \]

Here the average is over training images, random steps \(t\) and random noise. It is simply a mean squared error. For example, if the true noise in three pixels is \((0.2, -0.5, 0.7)\) and the model predicts \((0.1, -0.3, 0.6)\), the error is

\[ \frac{(0.1)^2 + (-0.2)^2 + (0.1)^2}{3} = \frac{0.01 + 0.04 + 0.01}{3} = 0.02. \]

Training nudges the network’s parameters to make this smaller, using gradient descent, the method that trains neural networks. Not every model predicts noise; some predict the clean image or a “velocity” instead, which are mathematically related.

Predicting the noise is closely related to learning the score of the noisy images at each step, \(\nabla_{x_t} \log p_t(x_t)\), an arrow pointing toward more likely images at that noise level. In DDPM it is approximately \(-\varepsilon_\theta(x_t, t)/\sqrt{1-\bar\alpha_t}\). Yang Song and colleagues (2021) showed that diffusion can be written as a stochastic differential equation, and generation follows the score back from noise to data.

Making a New Image (the Reverse Process)

  1. Start with pure noise: \(x_T \sim \mathcal{N}(0, I)\).
  2. Predict the noise \(\varepsilon_\theta(x_t, t)\) and use it to estimate a slightly cleaner \(x_{t-1}\).
  3. Add back a small amount of fresh randomness (in the original method), which keeps results varied.
  4. Repeat until \(t = 0\).

Because the starting noise is random, the same prompt gives a different image each time. Faster samplers such as DDIM (first posted in October 2020) cut 1,000 steps to a few dozen. The random “seed” sets the starting noise, so the same seed and settings reproduce the same image.

How the Text Prompt Steers the Picture

The prompt is turned into numbers (an embedding) by a text encoder, and the noise predictor receives it as extra input: \(\varepsilon_\theta(x_t, t, \text{prompt})\). A trick called classifier-free guidance (Ho and Salimans) predicts the noise twice, with and without the prompt, then exaggerates the difference:

\[ \hat\varepsilon = \varepsilon_\theta(x_t, \varnothing) + w\,\big(\varepsilon_\theta(x_t, \text{prompt}) – \varepsilon_\theta(x_t, \varnothing)\big), \]

where a guidance scale \(w\) above 1 makes images follow the prompt more closely, at some cost in variety.

How Does Stable Diffusion Work?

Classic Stable Diffusion (versions 1.x, based on latent diffusion by Rombach and colleagues, 2022) has four parts:

  1. Text encoder: turns the prompt into numbers (embeddings).
  2. U-Net noise predictor: predicts noise at each step, reading the prompt through cross-attention.
  3. Scheduler: uses the prediction to compute the next, slightly cleaner latent.
  4. VAE decoder: turns the final latent into a normal image.

The “latent” is the key speed-up. A 512 × 512 color image has 786,432 values; its Stable Diffusion 1.x latent is 64 × 64 × 4 = 16,384 values, so \(786{,}432 / 16{,}384 = 48\) times fewer numbers. That cuts the work of each denoising step a lot, though the whole pipeline is not exactly 48 times faster.

Beyond Classic Diffusion: Flow Matching and Other Methods

Newer models change the recipe. Flow matching (2022–2023) and rectified flow use a straight-line path between data and noise,

\[ x_t = (1 – t)\,x_0 + t\,\varepsilon, \qquad 0 \le t \le 1, \]

and train the network to predict the direction of travel, the velocity \(v = \varepsilon – x_0\). Generation then follows that learned direction from noise back to an image, often in fewer steps. Stable Diffusion 3 (2024) uses rectified flow with a transformer instead of a U-Net. Some other systems, such as OpenAI’s GPT-4o image generation, have been described as autoregressive, building images piece by piece more like text. Diffusion is one major family, not the only one.

A Short History

YearMilestone
2015Sohl-Dickstein and colleagues introduce diffusion models, inspired by non-equilibrium thermodynamics
2019–2021Yang Song and Stefano Ermon develop score-based models; Song and colleagues unify them with diffusion using stochastic differential equations (2021)
2020Ho, Jain and Abbeel’s DDPM shows diffusion can make high-quality images
2022Latent diffusion and Stable Diffusion; DALL·E 2 and Imagen bring text-to-image to the public
2022–2024Flow matching and rectified flow offer straighter paths from noise to image; Stable Diffusion 3 (2024) uses rectified flow

The Math You Need, Mapped

Math topicWhere it appears
Normal (Gaussian) distributionThe noise \(\varepsilon\) and the starting point \(x_T\)
Square roots and weighted averagesMixing image and noise in \(x_t\)
Products and sequences\(\bar\alpha_t = \prod (1 – \beta_s)\)
Mean squared errorThe training loss
Gradients and calculusThe score \(\nabla \log p\) and gradient-descent training
Linear algebraEvery layer of the neural network; see how linear algebra powers AI
Differential equationsContinuous-time diffusion and flow models (advanced)

For the probability side, see the probability behind AI.

Common Misconceptions

“Image generators copy and paste pieces of training images.” Not by design. They learn a statistical model of images and sample new ones, though they can sometimes reproduce images that appeared many times in training.

“The noise is just a starting placeholder.” No. The random starting noise determines the final image; change the noise and you change the picture.

“Every AI image generator uses the same 2020 algorithm.” No. Stable Diffusion 1.x, Stable Diffusion 3 and other systems use different networks, schedules and training targets, and some are not diffusion models at all.

“Diffusion is a physics simulation.” It borrows ideas and math from physics (diffusion of particles, thermodynamics), but it is a learned statistical method.

Watch: How Diffusion Models Work

If you learn better by watching, these videos are worth your time.

But how do AI images and videos actually work? | Guest video by Welch Labs (3Blue1Brown). A visual explanation of diffusion models, from noise to image.
Diffusion Models | Paper Explanation | Math Explained (Outlier). A walk through the DDPM paper and its equations.

Check Your Understanding

Five quick questions to see how much of this has stuck. Pick an answer and press Submit; you will see a short explanation either way.

Answer each question and press Submit to see your result right away.

1. What does a diffusion model learn to predict during training?
Found a bug? Let us know!
2. In x_t = √ᾱ·x_0 + √(1−ᾱ)·ε, what happens to ᾱ as t increases?
Found a bug? Let us know!
3. What does image generation start from?
Found a bug? Let us know!
4. Why does Stable Diffusion use a latent space?
Found a bug? Let us know!
5. What does classifier-free guidance do?
Found a bug? Let us know!

Frequently Asked Questions

How do AI image generators work?

Most use diffusion models: they start from random noise and repeatedly remove noise predicted by a neural network, guided by the text prompt, until an image appears.

What is the math behind diffusion models?

Gaussian noise is mixed with an image using x_t = √ᾱ_t · x_0 + √(1 − ᾱ_t) · ε, and a network is trained with mean squared error to predict the noise. Generation reverses the process step by step.

Why do diffusion models need so many steps?

Each step only removes a little noise; removing heavy noise in one jump is very inaccurate. Faster samplers reduce the original 1,000 steps to a few dozen.

What is Stable Diffusion?

A latent diffusion model first released in 2022. Versions 1.x run diffusion on a compressed 64 × 64 × 4 latent instead of 512 × 512 pixels, with a text encoder, a U-Net noise predictor and a decoder. Stable Diffusion 3 uses rectified flow.

What is the difference between diffusion and flow matching?

Classic diffusion trains a network to predict added noise; flow matching trains it to predict a velocity along a path from noise to data. Both generate images step by step.

Why does the same prompt give different images?

Each image starts from different random noise, and that noise shapes the final picture.

Keep Exploring

Sources and Further Reading

Related to This Article

What people say about "How AI Image Generators Work: Diffusion Math Explained - Effortless Math"?

No one replied yet.

Leave a Reply