# The Stable Artist: Steering Semantics in Diffusion Latent Space

arXiv: https://arxiv.org/abs/2212.06013 · HTML: https://mlsanity.com/paper/2212.06013/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 12 Dec 2022.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 11 of 11)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale](https://mlsanity.com/paper/2211.03759/) (2022)

## Built on by (most cited first, 23 of 23)

- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [In-Context Impersonation Reveals Large Language Models' Strengths and Biases](https://mlsanity.com/paper/2305.14930/) (2023)
- [DiT4Edit: Diffusion Transformer for Image Editing](https://mlsanity.com/paper/2411.03286/) (2024)
- [Controlling Language and Diffusion Models by Transporting Activations](https://mlsanity.com/paper/2410.23054/) (2024)
- [Style Vectors for Steering Generative Large Language Model](https://mlsanity.com/paper/2402.01618/) (2024)
- [The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion](https://mlsanity.com/paper/2309.04509/) (2023)
- [Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Models](https://mlsanity.com/paper/2306.05357/) (2023)
- [Stereotypes and Smut: The (Mis)representation of Non-cisgender Identities by Text-to-Image Models](https://mlsanity.com/paper/2305.17072/) (2023)
- [MDP: A Generalized Framework for Text-Guided Image Editing by Manipulating the Diffusion Path](https://mlsanity.com/paper/2303.16765/) (2023)
- [MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Mask](https://mlsanity.com/paper/2309.04399/) (2023)
- [Diffusion-Based Visual Art Creation: A Survey and New Perspectives](https://mlsanity.com/paper/2408.12128/) (2024)
- [Video Unlearning via Low-Rank Refusal Vector](https://mlsanity.com/paper/2506.07891/) (2025)
- [Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation](https://mlsanity.com/paper/2306.00964/) (2023)
- [Concept Arithmetics for Circumventing Concept Inhibition in Diffusion Models](https://mlsanity.com/paper/2404.13706/) (2024)
- [Safeguard Text-to-Image Diffusion Models with Human Feedback Inversion](https://mlsanity.com/paper/2407.21032/) (2024)
- [Data Redaction from Conditional Generative Models](https://mlsanity.com/paper/2305.11351/) (2023)
- [Zero-Shot Image Harmonization with Generative Model Prior](https://mlsanity.com/paper/2307.08182/) (2023)
- [Concept Steerers: Leveraging K-Sparse Autoencoders for Test-Time Controllable Generations](https://mlsanity.com/paper/2501.19066/) (2025)
- [Plug-and-Play Interpretable Responsible Text-to-Image Generation via Dual-Space Multi-facet Concept Control](https://mlsanity.com/paper/2503.18324/) (2025)
- [Positive-Unlabeled Diffusion Models for Preventing Sensitive Data Generation](https://mlsanity.com/paper/2503.03789/) (2025)
- ["My Kind of Woman": Analysing Gender Stereotypes in AI through The Averageness Theory and EU Law](https://mlsanity.com/paper/2407.17474/) (2024)
- [SHIFT: Steering Hidden Intermediates in Flow Transformers](https://mlsanity.com/paper/2604.09213/) (2026)

## Abstract (arXiv)

Large, text-conditioned generative diffusion models have recently gained a lot of attention for their impressive performance in generating high-fidelity images from text alone. However, achieving high-quality results is almost unfeasible in a one-shot fashion. On the contrary, text-guided image generation involves the user making many slight changes to inputs in order to iteratively carve out the envisioned image. However, slight changes to the input prompt often lead to entirely different images being generated, and thus the control of the artist is limited in its granularity. To provide flexibility, we present the Stable Artist, an image editing approach enabling fine-grained control of the image generation process. The main component is semantic guidance (SEGA) which steers the diffusion process along variable numbers of semantic directions. This allows for subtle edits to images, changes in composition and style, as well as optimization of the overall artistic conception. Furthermore, SEGA enables probing of latent spaces to gain insights into the representation of concepts learned by the model, even complex ones such as 'carbon emission'. We demonstrate the Stable Artist on several tasks, showcasing high-quality image editing and composition.