Inside Stable Diffusion

A note on how this was written. I put this post together in collaboration with AI agents. Using AI to work out how AI works turned out to be a genuinely good way to learn it — there is a longer note at the end.

You type a sentence and get a picture. In between are three networks and one loop that runs ten times.

Press play.

It works on a sketch

The surprise is that the U-Net never sees a picture. It works on a small stand-in — a 64 × 64 grid the model uses instead of the full image.

A 512 × 512 photo is 786,432 numbers. The sketch is 16,384. That is 48× smaller, and it is the whole reason this runs on a normal graphics card instead of a datacentre.

Only at the very end does the decoder blow the sketch up into pixels.

The loop

Each time round, two things happen:

1. The U-Net looks at the sketch and guesses which parts are noise.
2. The scheduler subtracts a little of that guess.

Repeat ten times and a picture falls out. The scheduler has no learned weights — it is arithmetic. All the intelligence is in the guess.

How much noise to remove is fixed in advance by a schedule. Drag the slider:

Notice how little the last few steps change. That is the shape of the schedule, and later versions fix it.

How the words get in

The text never enters as a picture-shaped thing. Instead, at every layer of the U-Net, each patch of the sketch asks the sentence a question: which words apply to me? The answer nudges that patch. This is cross-attention, and it is the only place the prompt touches anything.

Run it twice — once with your prompt, once with an empty one — and exaggerate the difference, and prompts bite harder. That is the “guidance scale” slider in every interface. It is also why each step costs two passes, not one.

Starting from a picture instead

Everything so far began with pure static. Feed the model a picture instead and almost nothing changes — which is the point.

The decoder you met at the end runs in reverse too. Push your image through it backwards and you get a sketch: the same 64 × 64 grid, only this one means something already. Add noise to it — but stop partway — and hand it to the same loop.

How far you noise it is the strength dial, the one every interface exposes and nobody explains. It decides where on the schedule you start:

Slide it all the way up and the input is gone — every trace of it has been noised away, and you are back to text-to-image. That is the whole relationship: text-to-image is image-to-image at strength 1. There is no second model, no second pipeline. Only a different place to start.

Slide it down and the loop has fewer steps left to run, which is why img2img at strength 0.4 is roughly half the wait.

	text → image	image → image
Encoder runs?	no	yes, once
Starts from	pure static	your picture, part-noised
Steps run	all 10	strength × 10
At strength 1	—	identical to text → image

Inpainting is the same trick with a mask: noise only the region you painted, leave the rest of the sketch alone, and the loop repairs the hole so that it agrees with its surroundings.

How it evolved

Four generations in, the only piece still recognisable is the sketch.

1.x is the baseline every diagram above describes: one CLIP text encoder, one U-Net, a 4-channel sketch at 512×512. 1.4 and 1.5 are the same architecture — 1.5 simply trained for longer, which is why it is the one that stuck and the one most fine-tunes still descend from.

2.x barely changed the shape. It swapped in a bigger text encoder and a steadier training target. It is remembered as a downgrade anyway — the training images had been filtered much more aggressively, and a better reader of thinner material still reads less.

SDXL made everything bigger rather than different: three times the U-Net, a second text encoder glued alongside the first, and 1024×1024 output. Same 4-channel sketch, same curved schedule. SDXL Turbo then distilled it down to one to four steps by training it against a discriminator.

3.x threw out the U-Net. In its place is a transformer that treats sketch patches and words as one long list and lets them attend to each other directly, instead of the prompt reaching in from the side. Two more things changed:

• A straight schedule. The old curve wasted its final steps. The new one moves at a constant rate, so fewer steps are needed.
• A richer sketch. Four channels became sixteen — four times the detail for the decoder to work with. That ceiling had been in place since 2022.

The models and their licences

A note on how this was written. I put this post together in collaboration with AI agents — drafting the explanations, building the diagrams, checking the arithmetic behind the schedule, and running down every licence and link in the table above. Using AI to work out how AI works turned out to be a genuinely good way to learn it: every hand-wave I tried to get away with had to be turned into something that actually rendered, or actually summed to 100%. The mistakes that survived are mine.