How to paint with noise

Here's a thing that sounds like it shouldn't be possible. You feed a model pure noise (the snow you'd get from an untuned TV) and out comes a sharp, detailed, never-before-seen photograph of a cat. Not retrieved, not collaged: generated, pixel by pixel, from static. The first time I really understood how, I laughed, because the trick is so sneaky it feels like cheating. The model never learns to paint. It learns to do something much easier, and then we run that easy thing backwards.

This is the other half of the modern AI story. I spent a whole post on transformers, the engine behind text. Diffusion is the engine behind images, and its central idea is completely different and, I think, even more surprising once it clicks.

Destroying an image is easy

Start at the easy end. Take an image and add a little random noise. Add a little more. Keep going, and after enough steps every trace of the original is gone and you're left with pure static. That's the forward process, and the lovely thing about it is that it requires no intelligence whatsoever. Anyone can destroy a picture. Drag the slider to ruin this one, or hit play to watch it dissolve.

At each step we mix a shrinking amount of the real image with a growing amount of fresh noise. The exact recipe, the thing the whole field is built on, is one line:

$$x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon, \qquad \varepsilon \sim \mathcal{N}(0, I)$$

Copy LaTeXwith $$ delimitersCopy mathexpression only

The single knob is $\bar\alpha_t$: at the start it's nearly 1, so the image is almost all signal; by the end it's nearly 0, so it's almost all noise. That trajectory from signal to noise is the schedule, and it's worth seeing on its own, because the reverse process is going to walk it backwards.

The two curves are how much signal and how much noise are in the picture at each step. They cross in the middle, where the image is half itself and half static, and because they're square-rooted shares of one whole, signal² plus noise² is always exactly 1. The image at any step is a pure blend, never anything more complicated.

Creating one is destruction, reversed

Now the sneaky part. We've got a process that turns images into noise. We want the opposite: noise into images. So we train a network to undo one step of the damage. Given a noisy image and which step it's at, it learns to predict the noise that was added. Subtract a bit of that predicted noise and you've got a slightly cleaner image. Do it again. And again. Start from pure static and run this denoiser all the way back, and an image you've never seen assembles itself out of the snow.

Because the forward process is fixed and known, the training data is free and infinite. Take any image, pick any noise level, add the corresponding noise, and you have a labelled example: the input is the noisy image and the step, the answer is the noise. The training objective is just "predict that noise well":

$$\mathcal{L} = \mathbb{E}_{x_0,\,t,\,\varepsilon}\left[\,\bigl\lVert \varepsilon - \varepsilon_\theta(x_t, t) \bigr\rVert^2\,\right]$$

Copy LaTeXwith $$ delimitersCopy mathexpression only

How it knows to draw a cat

If you just run the denoiser from random static you get a plausible image, but not one you asked for. To steer it, you feed in a condition, usually a text prompt, and the denoiser learns to predict the noise given that prompt. The text gets turned into a vector by a language encoder (which is, pleasingly, a transformer), and that vector nudges every denoising step toward images that match. A trick called classifier-free guidance lets you dial the strength of that nudge: turn it up and the model clings harder to your prompt, sometimes at the cost of looking a bit overcooked. So the two halves of modern AI meet here: a transformer reads your words, and a diffusion model paints them by un-noising.

What it costs

Nothing's free. The obvious price is speed: where a GAN generates an image in one forward pass, a diffusion model takes many denoising steps, dozens to a thousand depending on the sampler. A lot of recent research is about getting the step count down (better schedules, distillation, clever solvers) without wrecking quality. The schedule itself matters more than you'd guess: spend too few steps near the noisy end and the model never establishes the big shapes; too few near the clean end and the fine detail comes out mushy.

Some food for thought to end on. There's something almost philosophical about the idea that the way to create is to learn destruction and run it in reverse: that a detailed, specific image is just static with the right noise carefully removed. It rhymes with how sculptors talk about finding the statue inside the marble. I don't want to push the metaphor too far, but I do think it's one of those ideas that's prettier than it had any right to be.

Recap

A diffusion model is trained on the easiest possible task: look at a noisy image and guess the noise. Because the forward noising process is fixed, that training data is free and the task is well-posed. Chain the learned denoiser backwards from pure static, optionally steered by a text prompt, and an image emerges one small step at a time. Don't learn to paint. Learn to un-noise, then run it in reverse.

Reading further

• Ho, Jain & Abbeel (2020), Denoising Diffusion Probabilistic Models: the paper that made diffusion work and fixed the form of the equations above. arXiv:2006.11239
• Sohl-Dickstein et al. (2015): the original idea of learning to reverse a diffusion, five years before it caught fire. arXiv:1503.03585
• Song et al. (2021), Score-Based Generative Modeling through SDEs: the deeper view that unifies diffusion with score matching and continuous-time SDEs. arXiv:2011.13456
• Lilian Weng, What are Diffusion Models?: the best single explainer online once you want every term in the equations pinned down. lilianweng.github.io