What is Stable Diffusion

What it is

Stable Diffusion is a family of open-weights text-to-image models that generate detailed images from text descriptions, and also support image-to-image generation, inpainting, and outpainting. It was developed by Stability AI, originating from the latent diffusion project at LMU Munich's CompVis group and Runway, with compute donated by Stability AI and training data from the nonprofit LAION. Development was led by Patrick Esser and Robin Rombach. It was released on August 22, 2022.

Its significance was access. While competitors ran behind APIs, Stable Diffusion's weights were free to download and it ran on consumer GPUs, with optimized builds needing as little as 2.4GB of video memory. That single fact seeded an enormous ecosystem: web interfaces like AUTOMATIC1111 and ComfyUI, thousands of fine-tuned checkpoints for anime, photorealism, and every art style imaginable, and a community that treated the model as raw material rather than a finished product.

How it works

Stable Diffusion uses latent diffusion. A variational autoencoder compresses images into a smaller latent space, Gaussian noise is added iteratively, and then a neural network denoises that representation back into an image, guided by the text prompt through a CLIP text encoder connected via cross-attention. A decoder renders the final pixels. Working in latent space rather than pixel space is what made it fast enough for consumer hardware.

The first versions used a U-Net backbone at 860 million parameters with a 123-million-parameter text encoder, trained on LAION subsets at 512 by 512 resolution. Training took about 150,000 Nvidia A100 GPU-hours and cost roughly $600,000, a figure that became famous as evidence that frontier image models were within reach of well-funded startups.

The family grew quickly. SDXL arrived in July 2023 with a larger U-Net, two text encoders, multi-aspect-ratio training, and native 1024 resolution plus a refiner model for a second detail pass. SD 3 and 3.5, released through 2024, swapped the U-Net for a multimodal diffusion transformer using rectified flow; SD 3.5 Large has 8.1 billion parameters and Medium has 2.6 billion.

Fine-tuning methods let users personalize the model with a handful of images. DreamBooth teaches it a new subject, textual inversion teaches it new concepts as embeddings, and LoRA adapters add new styles in small downloadable files. The LoRA ecosystem alone produced tens of thousands of shared styles.

Key details

Weights are free to download; self-hosting costs only hardware. Licensing changed across versions, and the change matters. The 1.x, 2.x, and SDXL releases used the CreativeML Open RAIL-M license with no revenue cap, and outputs were commercially usable. SD 3 and 3.5 moved to the Stability AI Community License, which is free for research, non-commercial use, and commercial use by organizations under $1 million in annual revenue, with a paid enterprise license above that. Anyone building a business on the newer versions needs to read the license, not assume the old terms still apply. Hosted options like the Stability AI API exist for those who prefer not to run it themselves.

Limitations

The early versions had well-known weaknesses: human limbs and faces, legible text, and anything far from the 512-pixel training resolution came out mangled. SDXL improved on all three, and the 3.x generation improved text rendering further, but photorealistic hands remained a meme for a reason.

The training data skewed Western and English-captioned, so prompts defaulted to Western representations and the model performed worse in other languages, a bias the creators acknowledged in the model card. A broadcaster investigation found private and sensitive data inside LAION's datasets, including medical and personal imagery scraped from the web.

The legal exposure has been real and ongoing. Getty Images sued in London in January 2023 over allegedly scraped images; in November 2025 the court ruled the training itself was not secondary copyright infringement but found trademark infringement in versions 1.x and 2.1 for generating Getty-like watermarks. Three artists filed a class action in San Francisco in January 2023 against Stability AI, Midjourney, and DeviantArt. Open weights democratized the technology and distributed its controversies along with it, and the lawsuits are still shaping what training data the next generation of models can use.

The community that took over

Stable Diffusion's most remarkable feature was not in the model at all. It was the community that formed around the weights and, in many respects, took over development. Within weeks of release, the AUTOMATIC1111 web interface gave non-technical users a point-and-click front end with prompt weighting, inpainting, and batch generation. ComfyUI followed with a node-based workflow editor that treated image generation as a visual programming task, and it became the power user's tool of choice for complex pipelines.

Civitai emerged as the sharing hub: tens of thousands of checkpoints, LoRAs, and embeddings, rated and discussed like an app store for model components. The LoRA format was the key enabler. Small enough to share freely, LoRAs let anyone distribute a new style, character, or concept that others could plug into the base model. Entire aesthetic movements, pixel art, specific anime styles, architectural visualization looks, were packaged and passed around as files measured in megabytes.

Stability AI itself went through turbulence, with leadership changes and financial strain widely reported, but the ecosystem proved resilient because it no longer depended on the company. SDXL Turbo and later distilled variants brought generation down to near-real-time speeds. Commercial users built products on the RAIL-licensed versions while watching the 3.x license change carefully. The lesson of Stable Diffusion is that open weights create something no company can fully control: a technology whose future is decided by its users.