# Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models

arXiv: https://arxiv.org/abs/2312.04410 · HTML: https://mlsanity.com/paper/2312.04410/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 79 of its 97 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 79)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)

## Built on by (most cited first, 30 of 49)

- [SDXL-Lightning: Progressive Adversarial Diffusion Distillation](https://mlsanity.com/paper/2402.13929/) (2024)
- [Taming Rectified Flow for Inversion and Editing](https://mlsanity.com/paper/2411.04746/) (2024)
- [A Survey on LoRA of Large Language Models](https://mlsanity.com/paper/2407.11046/) (2024)
- [COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing](https://mlsanity.com/paper/2406.08850/) (2024)
- [Diffusion Models and Representation Learning: A Survey](https://mlsanity.com/paper/2407.00783/) (2024)
- [Parameter-Efficient Fine-Tuning for Foundation Models](https://mlsanity.com/paper/2501.13787/) (2025)
- [ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation](https://mlsanity.com/paper/2502.18364/) (2025)
- [Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediators](https://mlsanity.com/paper/2408.05710/) (2024)
- [Distilling Diversity and Control in Diffusion Models](https://mlsanity.com/paper/2503.10637/) (2025)
- [Diffusion Model with Perceptual Loss](https://mlsanity.com/paper/2401.00110/) (2023)
- [Training-Free Constrained Generation With Stable Diffusion Models](https://mlsanity.com/paper/2502.05625/) (2025)
- [FreeMorph: Tuning-Free Generalized Image Morphing with Diffusion Model](https://mlsanity.com/paper/2507.01953/) (2025)
- [PERSE: Personalized 3D Generative Avatars from A Single Portrait](https://mlsanity.com/paper/2412.21206/) (2024)
- [Alias-Free Latent Diffusion Models: Improving Fractional Shift Equivariance of Diffusion Latent Space](https://mlsanity.com/paper/2503.09419/) (2025)
- [GRA: Detecting Oriented Objects through Group-wise Rotating and Attention](https://mlsanity.com/paper/2403.11127/) (2024)
- [AdaNAT: Exploring Adaptive Policy for Token-Based Image Generation](https://mlsanity.com/paper/2409.00342/) (2024)
- [Probability Density Geodesics in Image Diffusion Latent Space](https://mlsanity.com/paper/2504.06675/) (2025)
- [Hessian Geometry of Latent Space in Generative Models](https://mlsanity.com/paper/2506.10632/) (2025)
- [PredBench: Benchmarking Spatio-Temporal Prediction across Diverse Disciplines](https://mlsanity.com/paper/2407.08418/) (2024)
- [Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials](https://mlsanity.com/paper/2511.00833/) (2025)
- [MRI Contrast Enhancement Kinetics World Model](https://mlsanity.com/paper/2602.19285/) (2026)
- [M3Face: A Unified Multi-Modal Multilingual Framework for Human Face Generation and Editing](https://mlsanity.com/paper/2402.02369/) (2024)
- [FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation](https://mlsanity.com/paper/2505.18494/) (2025)
- [IMG: Calibrating Diffusion Models via Implicit Multimodal Guidance](https://mlsanity.com/paper/2509.26231/) (2025)
- [Be Tangential to Manifold: Discovering Riemannian Metric for Diffusion Models](https://mlsanity.com/paper/2510.05509/) (2025)
- [Vibe Spaces for Creatively Connecting and Expressing Visual Concepts](https://mlsanity.com/paper/2512.14884/) (2025)
- [Redefining <Creative> in Dictionary: Towards an Enhanced Semantic Understanding of Creative Generation](https://mlsanity.com/paper/2410.24160/) (2024)
- [On the Fairness, Diversity and Reliability of Text-to-Image Generative Models](https://mlsanity.com/paper/2411.13981/) (2024)
- [Quantum Feature Space of a Qubit Coupled to an Arbitrary Bath](https://mlsanity.com/paper/2505.03397/) (2025)
- [SC-Pro: Training-Free Framework for Defending Unsafe Image Synthesis Attack](https://mlsanity.com/paper/2501.05359/) (2025)

## Abstract (arXiv)

Recently, diffusion models have made remarkable progress in text-to-image (T2I) generation, synthesizing images with high fidelity and diverse contents. Despite this advancement, latent space smoothness within diffusion models remains largely unexplored. Smooth latent spaces ensure that a perturbation on an input latent corresponds to a steady change in the output image. This property proves beneficial in downstream tasks, including image interpolation, inversion, and editing. In this work, we expose the non-smoothness of diffusion latent spaces by observing noticeable visual fluctuations resulting from minor latent variations. To tackle this issue, we propose Smooth Diffusion, a new category of diffusion models that can be simultaneously high-performing and smooth. Specifically, we introduce Step-wise Variation Regularization to enforce the proportion between the variations of an arbitrary input latent and that of the output image is a constant at any diffusion training step. In addition, we devise an interpolation standard deviation (ISTD) metric to effectively assess the latent space smoothness of a diffusion model. Extensive quantitative and qualitative experiments demonstrate that Smooth Diffusion stands out as a more desirable solution not only in T2I generation but also across various downstream tasks. Smooth Diffusion is implemented as a plug-and-play Smooth-LoRA to work with various community models. Code is available at https://github.com/SHI-Labs/Smooth-Diffusion.