# Boundary Guided Learning-Free Semantic Control with Diffusion Models

arXiv: https://arxiv.org/abs/2302.08357 · HTML: https://mlsanity.com/paper/2302.08357/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 56 of its 72 references that are in the map.
- First posted on arXiv 16 Feb 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 56)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)

## Built on by (most cited first, 23 of 23)

- [ROBIN: Robust and Invisible Watermarks for Diffusion Models with Adversarial Optimization](https://mlsanity.com/paper/2411.03862/) (2024)
- [D3: Scaling Up Deepfake Detection by Learning from Discrepancy](https://mlsanity.com/paper/2404.04584/) (2024)
- [Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions](https://mlsanity.com/paper/2403.17064/) (2024)
- [Interpreting the Weight Space of Customized Diffusion Models](https://mlsanity.com/paper/2406.09413/) (2024)
- [Exploring Low-Dimensional Subspaces in Diffusion Models for Controllable Image Editing](https://mlsanity.com/paper/2409.02374/) (2024)
- [Novel Object Synthesis via Adaptive Text-Image Harmony](https://mlsanity.com/paper/2410.20823/) (2024)
- [Influential Language Data Selection via Gradient Trajectory Pursuit](https://mlsanity.com/paper/2410.16710/) (2024)
- [Training-Free Style and Content Transfer by Leveraging U-Net Skip Connections in Stable Diffusion](https://mlsanity.com/paper/2501.14524/) (2025)
- [ERDDCI: Exact Reversible Diffusion via Dual-Chain Inversion for High-Quality Image Editing](https://mlsanity.com/paper/2410.14247/) (2024)
- [Implicit Bias Injection Attacks against Text-to-Image Diffusion Models](https://mlsanity.com/paper/2504.01819/) (2025)
- [Foundation Cures Personalization: Improving Personalized Models' Prompt Consistency via Hidden Foundation Knowledge](https://mlsanity.com/paper/2411.15277/) (2024)
- [Diff-GOn: Enhancing Diffusion Models for Goal-Oriented Communications](https://mlsanity.com/paper/2412.06980/) (2024)
- [The Linear Geometry of Interpretable Tokens: Jailbreaking Attacks and Defenses for Unlearned Diffusion Models](https://mlsanity.com/paper/2504.21307/) (2025)
- [TAIGen: Training-Free Adversarial Image Generation via Diffusion Models](https://mlsanity.com/paper/2508.15020/) (2025)
- [CASL: Concept-Aligned Sparse Latents for Interpreting Diffusion Models](https://mlsanity.com/paper/2601.15441/) (2026)
- [DETER: Detecting Edited Regions for Deterring Generative Manipulations](https://mlsanity.com/paper/2312.10539/) (2023)
- [ColorwAI: Generative Colorways of Textiles through GAN and Diffusion Disentanglement](https://mlsanity.com/paper/2407.11514/) (2024)
- [Unsupervised Region-Based Image Editing of Denoising Diffusion Models](https://mlsanity.com/paper/2412.12912/) (2024)
- [A Sampling-Based Domain Generalization Study with Diffusion Generative Models](https://mlsanity.com/paper/2310.09213/) (2023)
- [ReMix: Towards a Unified View of Consistent Character Generation and Editing](https://mlsanity.com/paper/2510.10156/) (2025)
- [FLUID: Training-Free Face De-identification via Latent Identity Substitution](https://mlsanity.com/paper/2511.17005/) (2025)
- [RemEdit: Efficient Diffusion Editing with Riemannian Geometry](https://mlsanity.com/paper/2601.17927/) (2026)
- [GASS: Geometry-Aware Spherical Sampling for Disentangled Diversity Enhancement in Text-to-Image Generation](https://mlsanity.com/paper/2602.17200/) (2026)

## Abstract (arXiv)

Applying pre-trained generative denoising diffusion models (DDMs) for downstream tasks such as image semantic editing usually requires either fine-tuning DDMs or learning auxiliary editing networks in the existing literature. In this work, we present our BoundaryDiffusion method for efficient, effective and light-weight semantic control with frozen pre-trained DDMs, without learning any extra networks. As one of the first learning-free diffusion editing works, we start by seeking a comprehensive understanding of the intermediate high-dimensional latent spaces by theoretically and empirically analyzing their probabilistic and geometric behaviors in the Markov chain. We then propose to further explore the critical step for editing in the denoising trajectory that characterizes the convergence of a pre-trained DDM and introduce an automatic search method. Last but not least, in contrast to the conventional understanding that DDMs have relatively poor semantic behaviors, we prove that the critical latent space we found already exhibits semantic subspace boundaries at the generic level in unconditional DDMs, which allows us to do controllable manipulation by guiding the denoising trajectory towards the targeted boundary via a single-step operation. We conduct extensive experiments on multiple DPMs architectures (DDPM, iDDPM) and datasets (CelebA, CelebA-HQ, LSUN-church, LSUN-bedroom, AFHQ-dog) with different resolutions (64, 256), achieving superior or state-of-the-art performance in various task scenarios (image semantic editing, text-based editing, unconditional semantic control) to demonstrate the effectiveness.