# Lipschitz Singularities in Diffusion Models

arXiv: https://arxiv.org/abs/2306.11251 · HTML: https://mlsanity.com/paper/2306.11251/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 32 of its 36 references that are in the map.
- First posted on arXiv 20 Jun 2023.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 32)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [Maximum Likelihood Training of Score-Based Diffusion Models](https://mlsanity.com/paper/2101.09258/) (2021)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [Simple diffusion: End-to-end diffusion for high resolution images](https://mlsanity.com/paper/2301.11093/) (2023)
- [Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.06503/) (2022)
- [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://mlsanity.com/paper/2305.08891/) (2023)
- [Perception Prioritized Training of Diffusion Models](https://mlsanity.com/paper/2204.00227/) (2022)
- [Score-Based Generative Modeling with Critically-Damped Langevin Diffusion](https://mlsanity.com/paper/2112.07068/) (2021)
- [Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation](https://mlsanity.com/paper/2106.05527/) (2021)
- [Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models](https://mlsanity.com/paper/2206.07309/) (2022)
- [Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order Denoising Score Matching](https://mlsanity.com/paper/2206.08265/) (2022)

## Built on by (most cited first, 21 of 21)

- [ScaleDreamer: Scalable Text-to-3D Synthesis with Asynchronous Score Distillation](https://mlsanity.com/paper/2407.02040/) (2024)
- [Unraveling the Smoothness Properties of Diffusion Models: A Gaussian Mixture Perspective](https://mlsanity.com/paper/2405.16418/) (2024)
- [Don't Start from Scratch: Behavioral Refinement via Interpolant-based Policy Diffusion](https://mlsanity.com/paper/2402.16075/) (2024)
- [Deep MMD Gradient Flow without adversarial training](https://mlsanity.com/paper/2405.06780/) (2024)
- [SADA: Stability-guided Adaptive Diffusion Acceleration](https://mlsanity.com/paper/2507.17135/) (2025)
- [On the Asymptotic Mean Square Error Optimality of Diffusion Models](https://mlsanity.com/paper/2403.02957/) (2024)
- [Hessian Geometry of Latent Space in Generative Models](https://mlsanity.com/paper/2506.10632/) (2025)
- [An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures](https://mlsanity.com/paper/2501.07930/) (2025)
- [Parallel Simulation for Log-concave Sampling and Score-based Diffusion Models](https://mlsanity.com/paper/2412.07435/) (2024)
- [Efficiently Access Diffusion Fisher: Within the Outer Product Span Space](https://mlsanity.com/paper/2505.23264/) (2025)
- [On Conditional Stochastic Interpolation for Generative Nonlinear Sufficient Dimension Reduction](https://mlsanity.com/paper/2512.18971/) (2025)
- [Solving Prior Distribution Mismatch in Diffusion Models via Optimal Transport](https://mlsanity.com/paper/2410.13431/) (2024)
- [VIPaint: Image Inpainting with Pre-Trained Diffusion Models via Variational Inference](https://mlsanity.com/paper/2411.18929/) (2024)
- [Optimal Stopping in Latent Diffusion Models](https://mlsanity.com/paper/2510.08409/) (2025)
- [Sharpen Your Flow: Sharpness-Aware Sampling for Flow Matching](https://mlsanity.com/paper/2605.11547/) (2026)
- [Dataset Distillation as Pushforward Optimal Quantization](https://mlsanity.com/paper/2501.07681/) (2025)
- [DistillKac: Few-Step Image Generation via Damped Wave Equations](https://mlsanity.com/paper/2509.21513/) (2025)
- [Expert-Data Alignment Governs Generation Quality in Decentralized Diffusion Models](https://mlsanity.com/paper/2602.02685/) (2026)
- [A Quantitative Approximation Framework for Flow Distillation in Diffusion Models](https://mlsanity.com/paper/2606.03820/) (2026)
- [Potential Score Matching: Debiasing Molecular Structure Sampling with Potential Energy Guidance](https://mlsanity.com/paper/2503.14569/) (2025)
- [Proof-of-Authorship for Diffusion-based AI Generated Content](https://mlsanity.com/paper/2603.17513/) (2026)

## Abstract (arXiv)

Diffusion models, which employ stochastic differential equations to sample images through integrals, have emerged as a dominant class of generative models. However, the rationality of the diffusion process itself receives limited attention, leaving the question of whether the problem is well-posed and well-conditioned. In this paper, we explore a perplexing tendency of diffusion models: they often display the infinite Lipschitz property of the network with respect to time variable near the zero point. We provide theoretical proofs to illustrate the presence of infinite Lipschitz constants and empirical results to confirm it. The Lipschitz singularities pose a threat to the stability and accuracy during both the training and inference processes of diffusion models. Therefore, the mitigation of Lipschitz singularities holds great potential for enhancing the performance of diffusion models. To address this challenge, we propose a novel approach, dubbed E-TSDM, which alleviates the Lipschitz singularities of the diffusion model near the zero point of timesteps. Remarkably, our technique yields a substantial improvement in performance. Moreover, as a byproduct of our method, we achieve a dramatic reduction in the Fr'echet Inception Distance of acceleration methods relying on network Lipschitz, including DDIM and DPM-Solver, by over 33%. Extensive experiments on diverse datasets validate our theory and method. Our work may advance the understanding of the general diffusion process, and also provide insights for the design of diffusion models.