mlsanityData to 7 Oct 2026

A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data

arXiv 2402.16991 ↗First posted 26 Feb 2024stat.ML cond-mat.dis-nn cs.CVcited by 88

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Understanding the structure of real data is paramount in advancing modern deep-learning methodologies. Natural data such as images are believed to be composed of features organized in a hierarchical and combinatorial manner, which neural networks capture during learning. Recent advancements show that diffusion models can generate high-quality images, hinting at their ability to capture this underlying compositional structure. We study this phenomenon in a hierarchical generative model of data. We find that the backward diffusion process acting after a time t is governed by a phase transition at some threshold time, where the probability of reconstructing high-level features, like the class of an image, suddenly drops. Instead, the reconstruction of low-level features, such as specific details of an image, evolves smoothly across the whole diffusion process. This result implies that at times beyond the transition, the class has changed, but the generated sample may still be composed of low-level elements of the initial image. We validate these theoretical insights through numerical experiments on class-unconditional ImageNet diffusion models. Our analysis characterizes the relationship between time and scale in diffusion models and puts forward generative models as powerful tools to model combinatorial data properties.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 34 of its 70 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

34 of 70 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability26 Oct
  2. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task13 Oct
  3. 2023Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Models20 Sep
  4. 2023Learning Mixtures of Gaussians Using the DDPM Objective3 Jul
  5. 2023Spontaneous Symmetry Breaking in Generative Diffusion Models31 May
  6. 2023Diffusion Models are Minimax Optimal Distribution Estimators3 Mar
  7. 2023Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data14 Feb
  8. 2022Convergence of denoising diffusion models under the manifold hypothesis10 Aug
  9. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  10. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  11. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  12. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
1 more
  1. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  4. 2022A ConvNet for the 2020scited by 3,545
  5. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  6. 2023Extracting Training Data from Diffusion Modelscited by 723
  7. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  8. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  9. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  10. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  11. 2020Generative Modeling with Denoising Auto-Encoders and Langevin Samplingcited by 118
  12. 2017Failures of Gradient-Based Deep Learningcited by 79
9 more
  1. 2021Unsupervised Object-Level Representation Learning from Scene Imagescited by 72
  2. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
  3. 2020Self-Supervised Visual Representation Learning from Hierarchical Groupingcited by 63
  4. 2023Reward-Directed Conditional Diffusion: Provable Distribution Estimation and Reward Improvementcited by 49
  5. 2023Analysis of learning a flow-based generative model from limited sample complexitycited by 38
  6. 2021Locality defeats the curse of dimensionality in convolutional teacher-student scenarioscited by 26
  7. 2016Deep Learning and Hierarchal Generative Modelscited by 23
  8. 2023Learning multi-scale local conditional probability models of imagescited by 21
  9. 2018A Provably Correct Algorithm for Deep Learning that Actually Workscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 30; 2026: 41.

88 papers.

Sort:

By area:

  1. 2024An analytic theory of creativity in convolutional diffusion modelscited by 103
  2. 2025On the Closed-Form of Flow Matching: Generalization Does Not Arise from Target Stochasticitycited by 48
  3. 2024Critical windows: non-asymptotic theory for feature emergence in diffusion modelscited by 41
  4. 2024Towards a theory of how the structure of language is acquired by deep neural networkscited by 38
  5. 2024Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusioncited by 31
  6. 2024Dynamic Negative Guidance of Diffusion Modelscited by 28
  7. 2024Probing the Latent Hierarchical Structure of Data via Diffusion Modelscited by 25
  8. 2025How Compositional Generalization and Creativity Improve as Diffusion Models are Trainedcited by 24
  9. 2025Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Modelscited by 24
  10. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  11. 2025Denoising Score Matching with Random Features: Insights on Diffusion Models from Precise Learning Curvescited by 20
  12. 2024On the Feature Learning in Diffusion Modelscited by 19
38 more
  1. 2024How transformers learn structured data: insights from hierarchical filteringcited by 18
  2. 2025Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modelingcited by 18
  3. 2024U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Modelscited by 17
  4. 2024Learning Discrete Concepts in Latent Hierarchical Modelscited by 17
  5. 2025Learning curves theory for hierarchically compositional data with power-law distributed featurescited by 17
  6. 2025Feedback Guidance of Diffusion Modelscited by 16
  7. 2025A solvable model of learning generative diffusion: theory and insightscited by 14
  8. 2025CoDe: Blockwise Control for Denoising Diffusion Modelscited by 14
  9. 2025Entropic Time Schedulers for Generative Diffusion Modelscited by 13
  10. 2024Cascade of phase transitions in the training of Energy-based modelscited by 12
  11. 2025Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Facecited by 12
  12. 2025The Information Dynamics of Generative Diffusioncited by 12
  13. 2025Analysis of Diffusion Models for Manifold Datacited by 11
  14. 2024On learning higher-order cumulants in diffusion modelscited by 10
  15. 2024Avoiding mode collapse in diffusion models fine-tuned with reinforcement learningcited by 9
  16. 2025On the Emergence of Linear Analogies in Word Embeddingscited by 9
  17. 2025Hessian Geometry of Latent Space in Generative Modelscited by 9
  18. 2025Neural cellular automata: applications to biology and beyond classical AIcited by 9
  19. 2026Video Models Reason Early: Exploiting Plan Commitment for Maze Solvingcited by 9
  20. 2025Local Diffusion Models and Phases of Data Distributionscited by 8
  21. 2024How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Modelcited by 7
  22. 2026Theory of Speciation Transitions in Diffusion Models with General Class Structurecited by 7
  23. 2026The Entropic Signature of Class Speciation in Diffusion Modelscited by 7
  24. 2025Optimizing Noise Schedules of Generative Models in High Dimensionsscited by 6
  25. 2025The Generation Phases of Flow Matching: a Denoising Perspectivecited by 6
  26. 2025Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architecturescited by 5
  27. 2025Measuring Semantic Information Production in Generative Diffusion Modelscited by 5
  28. 2025Concept Unlearning by Modeling Key Steps of Diffusion Processcited by 5
  29. 2026How Out-of-Equilibrium Phase Transitions can Seed Pattern Formation in Trained Diffusion Modelscited by 5
  30. 2024Latent Abstractions in Generative Diffusion Modelscited by 4
  31. 2026Deep networks learn to parse uniform-depth context-free languages from local statisticscited by 4
  32. 2026Noise Scheduling as Information-Guided Allocation in Diffusion Trainingcited by 4
  33. 2025Local Mechanisms of Compositional Generalization in Conditional Diffusioncited by 3
  34. 2026Towards Worst-Case Guarantees with Scale-Aware Interpretabilitycited by 3
  35. 2026Steering Dynamical Regimes of Diffusion Models by Breaking Detailed Balancecited by 3
  36. 2026Biased Generalization in Diffusion Modelscited by 3
  37. 2025Not Every Time and Frequency Need to Be Forgotten in Diffusion Unlearningcited by 2
  38. 2026Rethinking Cross-Layer Information Routing in Diffusion Transformerscited by 2

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Let us Build Bridges: Understanding and Extending Diffusion Generative Models
  2. 2022Diffusion Models: A Comprehensive Survey of Methods and Applications
  3. 2024Dynamical Regimes of Diffusion Models
  4. 2024Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributions
  5. 2025Accelerating Diffusion Model Training under Minimal Budgets: A Condensation-Based Perspective
  6. 2024Non-Normal Diffusion Models
  7. 2024Tracing the Roots: Leveraging Temporal Dynamics in Diffusion Trajectories for Origin Attribution
  8. 2024GUD: Generation with Unified Diffusion