mlsanityData to 14 Oct 2026

An analytic theory of creativity in convolutional diffusion models

arXiv 2412.20292 ↗First posted 28 Dec 2024cs.LG cond-mat.dis-nn cs.AIcited by 104

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

We obtain an analytic, interpretable and predictive theory of creativity in convolutional diffusion models. Indeed, score-matching diffusion models can generate highly original images that lie far from their training data. However, optimal score-matching theory suggests that these models should only be able to produce memorized training examples. To reconcile this theory-experiment gap, we identify two simple inductive biases, locality and equivariance, that: (1) induce a form of combinatorial creativity by preventing optimal score-matching; (2) result in fully analytic, completely mechanistically interpretable, local score (LS) and equivariant local score (ELS) machines that, (3) after calibrating a single time-dependent hyperparameter can quantitatively predict the outputs of trained convolution only diffusion models (like ResNets and UNets) with high accuracy (median r2 of 0.95, 0.94, 0.94, 0.96 for our top model on CIFAR10, FashionMNIST, MNIST, and CelebA). Our model reveals a locally consistent patch mosaic mechanism of creativity, in which diffusion models create exponentially many novel images by mixing and matching different local training set patches at different scales and image locations. Our theory also partially predicts the outputs of pre-trained self-attention enabled UNets (median r2 ~ 0.77 on CIFAR10), revealing an intriguing role for attention in carving out semantic coherence from local patch mosaics.

Key facts. Cited by 104 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 44 of its 51 references that are in the map. First posted on arXiv 28 Dec 2024.

Builds on

44 of 51 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Detecting Multimedia Generated by Large AI Models: A Survey22 Jan
  2. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models7 Dec
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2022Scalable Diffusion Models with Transformerscited by 6,054
  10. 2022Flow Matching for Generative Modelingcited by 5,702
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
29 more
  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  2. 2022Video Diffusion Modelscited by 2,305
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  4. 2016Group Equivariant Convolutional Networkscited by 1,165
  5. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  6. 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 677
  7. 2022Convergence of denoising diffusion models under the manifold hypothesiscited by 242
  8. 2023Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localizationcited by 235
  9. 2023Generalization in diffusion models arises from geometry-adaptive harmonic representationscited by 185
  10. 2022Convergence for score-based generative modeling with polynomial complexitycited by 177
  11. 2023Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Datacited by 174
  12. 2023Diffusion Models are Minimax Optimal Distribution Estimatorscited by 165
  13. 2024Dynamical Regimes of Diffusion Modelscited by 142
  14. 2023On Memorization in Diffusion Modelscited by 125
  15. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 93
  16. 2024A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Datacited by 93
  17. 2024Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidancecited by 59
  18. 2023Diffusion Models Generate Images Like Painters: an Analytical Theory of Outline First, Details Latercited by 55
  19. 2024Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributionscited by 53
  20. 2023Closed-Form Diffusion Modelscited by 46
  21. 2023Diffusion Models for Non-autoregressive Text Generation: A Surveycited by 44
  22. 2023In search of dispersed memories: Generative diffusion models are associative memory networkscited by 44
  23. 2024Towards a Mechanistic Explanation of Diffusion Model Generalizationcited by 42
  24. 2023Analysis of learning a flow-based generative model from limited sample complexitycited by 38
  25. 2024Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusioncited by 32
  26. 2024The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applicationscited by 32
  27. 2024Diffusion Models in textitDe Novo Drug Designcited by 26
  28. 2023Learning multi-scale local conditional probability models of imagescited by 21
  29. 2023High-dimensional Asymptotics of Denoising Autoencoderscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 54; 2026: 50.

104 papers.

Sort:

By area:

  1. 2025Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Trainingcited by 81
  2. 2025On the Closed-Form of Flow Matching: Generalization Does Not Arise from Target Stochasticitycited by 50
  3. 2025Memorization to Generalization: Emergence of Diffusion Models from Associative Memorycited by 36
  4. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 29
  5. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  6. 2025An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Modelscited by 26
  7. 2025How Compositional Generalization and Creativity Improve as Diffusion Models are Trainedcited by 25
  8. 2025Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Modelscited by 25
  9. 2025Locality in Image Diffusion Models Emerges from Data Statisticscited by 25
  10. 2025Generalization through variance: how noise shapes inductive biases in diffusion modelscited by 23
  11. 2025On the Edge of Memorization in Diffusion Modelscited by 22
  12. 2025Denoising Score Matching with Random Features: Insights on Diffusion Models from Precise Learning Curvescited by 20
38 more
  1. 2025Diffusion Models and the Manifold Hypothesis: Log-Domain Smoothing is Geometry Adaptivecited by 20
  2. 2025Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modelingcited by 18
  3. 2025Ambient Diffusion Omni: Training Good Models with Bad Datacited by 18
  4. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspectivecited by 18
  5. 2025Selective Underfitting in Diffusion Modelscited by 18
  6. 2025Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusioncited by 17
  7. 2025On the Interpolation Effect of Score Smoothing in Diffusion Modelscited by 15
  8. 2025A solvable model of learning generative diffusion: theory and insightscited by 14
  9. 2025Mechanisms of Projective Composition of Diffusion Modelscited by 14
  10. 2025Provable Separations between Memorization and Generalization in Diffusion Modelscited by 14
  11. 2025Flow Map Distillation Without Datacited by 14
  12. 2025Density Ratio Estimation with Conditional Probability Pathscited by 13
  13. 2026Making Reconstruction FID Predictive of Diffusion Generation FIDcited by 13
  14. 2025Generalization of Diffusion Models Arises with a Balanced Representation Spacecited by 11
  15. 2025Resolving Memorization in Empirical Diffusion Model for Manifold Data in High-Dimensional Spacescited by 8
  16. 2025Local Diffusion Models and Phases of Data Distributionscited by 8
  17. 2025From Navigation to Refinement: Revealing the Two-Stage Nature of Flow-based Diffusion Models through Oracle Velocitycited by 8
  18. 2025The Generation Phases of Flow Matching: a Denoising Perspectivecited by 7
  19. 2026Revisiting Diffusion Model Predictions Through Dimensionalitycited by 7
  20. 2026A Random Matrix Theory Perspective on the Consistency of Diffusion Modelscited by 6
  21. 2026How Out-of-Equilibrium Phase Transitions can Seed Pattern Formation in Trained Diffusion Modelscited by 6
  22. 2025Localized Diffusion Modelscited by 5
  23. 2025Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformercited by 5
  24. 2026Diffusion Model's Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifoldcited by 5
  25. 2026Local Coverage Governs Memorization in Diffusion Modelscited by 5
  26. 2025Compositional Generalization via Forced Rendering of Disentangled Latentscited by 4
  27. 2025Geometric Regularity in Deterministic Sampling Dynamics of Diffusion-based Generative Modelscited by 4
  28. 2025Origins of Creativity in Attention-Based Diffusion Modelscited by 4
  29. 2025Local Learning Rules for Out-of-Equilibrium Physical Generative Modelscited by 4
  30. 2025Nonparametric Data Attribution for Diffusion Modelscited by 4
  31. 2025Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rulescited by 4
  32. 2026Biased Generalization in Diffusion Modelscited by 4
  33. 2025Predicting Forced Responses of Probability Distributions via the Fluctuation-Dissipation Theorem and Generative Modelingcited by 3
  34. 2025Kernel-Smoothed Scores for Denoising Diffusion: A Bias-Variance Studycited by 3
  35. 2025Local Mechanisms of Compositional Generalization in Conditional Diffusioncited by 3
  36. 2025Score-based Membership Inference on Diffusion Modelscited by 3
  37. 2025What Drives Compositional Generalization in Visual Generative Models? The Importance of Continuous Training Objectivescited by 3
  38. 2026Two Calm Ends and the Wild Middle: A Geometric Picture of Memorization in Diffusion Modelscited by 3

50 most cited of 104 All 104 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Elucidating the Design Space of Diffusion-Based Generative Models
  2. 2022A Survey on Generative Diffusion Model
  3. 2026Creative Image Generation with Diffusion Models
  4. 2025Enhancing Creative Generation on Stable Diffusion-based Models
  5. 2024Latent Representation Matters: Human-like Sketches in One-shot Drawing Tasks
  6. 2023Measuring the Success of Diffusion Models at Imitating Human Artists
  7. 2025Distribution-Conditional Generation: From Class Distribution to Creative Generation
  8. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization