ControlNet
Spatial conditioning controls for Stable Diffusion: edges, depth, pose, and more.
About
ControlNet is a neural network architecture that adds spatial conditioning controls to large pretrained text-to-image diffusion models. It locks the production-ready diffusion backbone and learns conditioning controls such as Canny edges, depth maps, human pose, segmentation, and scribbles through zero-initialized convolution layers, so training adds no harmful noise to the base model.
Introduced in the 2023 paper by Lvmin Zhang, Anyi Rao, and Maneesh Agrawala, it became the standard way to get precise compositional control over Stable Diffusion generations. The reference implementation lives at lllyasviel/ControlNet, and official checkpoints are distributed through Hugging Face. It is widely used inside Automatic1111, ComfyUI, and the diffusers library.
Practical notes: ControlNet models are separate checkpoints downloaded per control type, and they add VRAM cost on top of the base diffusion model. Quality depends on the preprocessor output (pose detectors, depth estimators), and like all image synthesis tooling it can be misused for deceptive imagery.
What it can do
- Edge-conditioned generation (Canny, HED)
- Depth and normal-map control
- OpenPose human-pose conditioning
- Segmentation and scribble guidance
- Multi-ControlNet combined conditioning
Setup
Clone lllyasviel/ControlNet or install via the ControlNet extension in Automatic1111 / ComfyUI; download the control-type checkpoints you need from Hugging Face.
Prerequisites
A local GPU with enough VRAM and a Stable Diffusion base model.
Use cases
Works with
Trust score — 9/10
Scored 9/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 87; Trending score 88/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.