mlsanityData to 7 Oct 2026

A solvable model of learning generative diffusion: theory and insights

arXiv 2501.03937 ↗First posted 7 Jan 2025cs.LG cond-mat.dis-nncited by 14

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

In this manuscript, we consider the problem of learning a flow or diffusion-based generative model parametrized by a two-layer auto-encoder, trained with online stochastic gradient descent, on a high-dimensional target density with an underlying low-dimensional manifold structure. We derive a tight asymptotic characterization of low-dimensional projections of the distribution of samples generated by the learned model, ascertaining in particular its dependence on the number of training samples. Building on this analysis, we discuss how mode collapse can arise, and lead to model collapse when the generative model is re-trained on generated synthetic data.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 70 of its 110 references that are in the map. First posted on arXiv 7 Jan 2025.

Builds on

70 of 110 references.

Closest ancestors · 37

Same direction, newest first

  1. 2024Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure31 Oct
  2. 2024Probing the Latent Hierarchical Structure of Data via Diffusion Models17 Oct
  3. 2024Shallow diffusion networks provably learn hidden low-dimensional structure15 Oct
  4. 2024Losing dimensions: Geometric memorization in generative diffusion11 Oct
  5. 2024Convergence of Diffusion Models Under the Manifold Hypothesis in High-Dimensions27 Sep
  6. 2024From optimal score matching to optimal sampling11 Sep
  7. 2024Replication in Visual Diffusion Models: A Survey and Outlook7 Jul
  8. 2024Evaluating the design space of diffusion-based generative models18 Jun
  9. 2024Adapting to Unknown Low-Dimensional Structures in Score-Based Diffusion Models23 May
  10. 2024Learning general Gaussian mixtures with efficient score matching29 Apr
  11. 2024Learning Mixtures of Gaussians Using Diffusion Models29 Apr
  12. 2024U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Models29 Apr
25 more
  1. 2024Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models3 Mar
  2. 2024Critical windows: non-asymptotic theory for feature emergence in diffusion models3 Mar
  3. 2024Dynamical Regimes of Diffusion Models28 Feb
  4. 2024A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data26 Feb
  5. 2024Minimax Optimality of Score-based Diffusion Models: Beyond the Density Lower Bound Assumptions23 Feb
  6. 2024Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization28 Jan
  7. 2023On the Generalization Properties of Diffusion Models3 Nov
  8. 2023Generalization in diffusion models arises from geometry-adaptive harmonic representations4 Oct
  9. 2023Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localization7 Aug
  10. 2023Towards Faster Non-Asymptotic Convergence for Diffusion-Based Generative Models15 Jun
  11. 2023Error Bounds for Flow Matching Methods26 May
  12. 2023The probability flow ODE is provably fast19 May
  13. 2023Diffusion Models Generate Images Like Painters: an Analytical Theory of Outline First, Details Later4 Mar
  14. 2023Diffusion Models are Minimax Optimal Distribution Estimators3 Mar
  15. 2023Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data14 Feb
  16. 2022Improved Analysis of Score-based Generative Modeling: User-Friendly Bounds under Minimal Smoothness Assumptions3 Nov
  17. 2022Convergence of score-based generative modeling for general data distributions26 Sep
  18. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions22 Sep
  19. 2022Convergence of denoising diffusion models under the manifold hypothesis10 Aug
  20. 2022Convergence for score-based generative modeling with polynomial complexity13 Jun
  21. 2022Score-Based Generative Models Detect Manifolds2 Jun
  22. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  23. 2021Heavy-tailed denoising score matching17 Dec
  24. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  25. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2022Flow Matching for Generative Modelingcited by 5,426
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  9. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  10. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  11. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
  12. 2015Statistical physics of inference: Thresholds and algorithmscited by 128
21 more
  1. 2020Generative Modeling with Denoising Auto-Encoders and Langevin Samplingcited by 118
  2. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'cited by 114
  3. 2024An analytic theory of creativity in convolutional diffusion modelscited by 103
  4. 2020Solving Linear Inverse Problems Using the Prior Implicit in a Denoisercited by 78
  5. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  6. 2022Statistical Efficiency of Score Matching: The View from Isoperimetrycited by 63
  7. 2024Optimal score estimation via empirical Bayes smoothingcited by 60
  8. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  9. 2024A Geometric Framework for Understanding Memorization in Generative Modelscited by 40
  10. 2023Analysis of learning a flow-based generative model from limited sample complexitycited by 38
  11. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  12. 2022The dynamics of representation learning in shallow, non-linear autoencoderscited by 27
  13. 2023Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspectivecited by 26
  14. 2023High-dimensional Asymptotics of Denoising Autoencoderscited by 19
  15. 2024How Do Flow Matching Models Memorize and Generalize in Sample Data Subspaces?cited by 19
  16. 2018Learning Dynamics of Linear Denoising Autoencoderscited by 16
  17. 2019Mean-field inference methods for neural networkscited by 16
  18. 2023High-dimensional robust regression under heavy-tailed data: Asymptotics and Universalitycited by 15
  19. 2024A theoretical perspective on mode collapse in variational inferencecited by 14
  20. 2024High-dimensional learning of narrow neural networkscited by 12
  21. 2023Classification of Heavy-tailed Features in High Dimensions: a Superstatistical Approachcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 10.

14 papers.

Sort:

By area:

  1. 2025Low-dimensional adaptation of diffusion models: Convergence in total variationcited by 38
  2. 2025Memorization to Generalization: Emergence of Diffusion Models from Associative Memorycited by 35
  3. 2026A theory of learning data statistics in diffusion models, from easy to hardcited by 5
  4. 2025Kernel-Smoothed Scores for Denoising Diffusion: A Bias-Variance Studycited by 3
  5. 2025A statistical physics framework for optimal learningcited by 3
  6. 2026Quantifying Error Propagation and Model Collapse in Diffusion Modelscited by 3
  7. 2026An exact information theory of generalization phase transitions in Bayesian diffusion modelscited by 3
  8. 2026Position: the Stochastic Parrot in the Coal Mine. Model Collapse is a Threat to Low-Resource Communitiescited by 1
  9. 2026The Interplay of Data Structure and Imbalance in the Learning Dynamics of Diffusion Modelscited by 1
  10. 2026Double Descent and Malign Overfitting in Diffusion Modelscited by 1
  11. 2026Diffusion Image Generation with Explicit Modeling of Data Manifold Geometrycited by 0
  12. 2026Recursively Trained Diffusion Models: Limiting Collapse Distribution and Spectral Characterizationcited by 0
2 more
  1. 2026Hierarchical Prototype Emergence in Modern Hopfield Modelscited by 0
  2. 2026Weighting Schedules Govern What and When Score-Based Generative Models Learn from Multimodal Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion Models: A Mathematical Introduction
  2. 2022From Points to Functions: Infinite-dimensional Representations in Diffusion Models
  3. 2022Understanding Diffusion Models: A Unified Perspective
  4. 2022Let us Build Bridges: Understanding and Extending Diffusion Generative Models
  5. 2023Diffusion Models for Generative Artificial Intelligence: An Introduction for Applied Mathematicians
  6. 2024Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributions
  7. 2022Truncated Diffusion Probabilistic Models and Diffusion-based Adversarial Auto-Encoders
  8. 2024GUD: Generation with Unified Diffusion