Conditioning
Telling a generative model what to make — and the difference between a slot machine and a tool.
When not to use it
- When you want variety. Conditioning constrains the distribution by definition. Heavy control means samey output.
- At high conditioning strength, reflexively. Rigid, artefact-laden results. The scale is a dial, not a switch.
- Text prompts, for composition. They're the weakest control. Sketch it and condition on the sketch.
- When exploring. Early on you want the model's ideas, not yours. Condition later.
Reach for something else instead
- Image-to-image — simpler than structural conditioning and often enough.
- Inpainting — when you only need part of the image controlled.
- Fine-tuning / LoRA — when the thing you want controlled is a subject or style, not a structure.
- Just drawing it — sometimes the control you need is a pencil.
Sources & further reading
- Zhang, Rao & Agrawala (2023), Adding Conditional Control to Text-to-Image Diffusion Models — ControlNet; zero-initialised injection into a frozen base.
- Ho & Salimans (2022), Classifier-Free Diffusion Guidance — the mechanism behind the guidance scale you've been tuning.
- Dhariwal & Nichol (2021), Diffusion Models Beat GANs on Image Synthesis — classifier guidance; where the trade between fidelity and diversity got made explicit.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Fighting the prompt for composition. Prompts describe; they don't specify. Use a structural condition.
- Maxing the conditioning scale. You get rigidity and artefacts, not obedience.
- Blaming the generator for failed spatial relations. That's the text encoder losing the relations.
- Expecting identity consistency from prompting. It's the hardest open problem in this area.
At a glance
FieldGenerative AI
What it ismodelling p(x|c) rather than p(x)
Control hierarchytext < image-to-image < structural < identity
Key mechanismclassifier-free guidance, an extrapolation
ControlNet's trickzero-initialised injection, so the base can't break
The real frontiercontrol, not quality
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Prompt vs. structural conditioning — one describes and hopes; the other fixes the composition and lets the prompt handle appearance. Ten seconds of sketching beats an hour of prompt archaeology.