mlsanityData to 7 Oct 2026

Information-Theoretic Diffusion

arXiv 2302.03792 ↗First posted 7 Feb 2023cs.LG cs.IT math.ITcited by 41

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Denoising diffusion models have spurred significant gains in density modeling and image generation, precipitating an industrial revolution in text-guided AI art generation. We introduce a new mathematical foundation for diffusion models inspired by classic results in information theory that connect Information with Minimum Mean Square Error regression, the so-called I-MMSE relations. We generalize the I-MMSE relations to exactly relate the data distribution to an optimal denoising regression problem, leading to an elegant refinement of existing diffusion bounds. This new insight leads to several improvements for probability distribution estimation, including theoretical justification for diffusion model ensembling. Remarkably, our framework shows how continuous and discrete probabilities can be learned with the same regression objective, avoiding domain-specific generative models used in variational methods. Code to reproduce experiments is provided at http://github.com/kxh001/ITdiffusion and simplified demonstration code is at http://github.com/gregversteeg/InfoDiffusionSimple.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 24 of its 49 references that are in the map. First posted on arXiv 7 Feb 2023.

Builds on

24 of 49 references.

Closest ancestors · 12

Same direction, newest first

  1. 2022Let us Build Bridges: Understanding and Extending Diffusion Generative Models31 Aug
  2. 2022Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise19 Aug
  3. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning8 Aug
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  5. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesis29 Nov
  6. 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul
  7. 2021Variational Diffusion Models1 Jul
  8. 2021A Variational Perspective on Diffusion-Based Generative Models and Score Matching5 Jun
  9. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  10. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  11. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  12. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 12

  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  5. 2018Neural Ordinary Differential Equationscited by 3,733
  6. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  7. 2019Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Designcited by 258
  8. 2019The Thermodynamic Variational Objectivecited by 26
  9. 2021q-Paths: Generalizing the Geometric Annealing Path using Power Meanscited by 12
  10. 2020All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inferencecited by 11
  11. 2021Variational Inference with Holder Boundscited by 3
  12. 2013Generalized Bregman Divergence and Gradient of Mutual Information for Vector Poisson Channelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 15; 2026: 17.

41 papers.

Sort:

By area:

  1. 2023Interpretable Diffusion via Information Decompositioncited by 37
  2. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 33
  3. 2023MINDE: Mutual Information Neural Diffusion Estimationcited by 31
  4. 2025Maximum Entropy Reinforcement Learning with Diffusion Policycited by 29
  5. 2024CPR: Retrieval Augmented Generation for Copyright Protectioncited by 22
  6. 2025Learning normalized image densities via dual score matchingcited by 17
  7. 2023Training Data Protection with Compositional Diffusion Modelscited by 16
  8. 2024Understanding Diffusion Models by Feynman's Path Integralcited by 13
  9. 2025Entropic Time Schedulers for Generative Diffusion Modelscited by 13
  10. 2025The Information Dynamics of Generative Diffusioncited by 12
  11. 2024Neural Entropycited by 7
  12. 2026The Entropic Signature of Class Speciation in Diffusion Modelscited by 7
29 more
  1. 2025Information-Theoretic Proofs for Diffusion Samplingcited by 6
  2. 2024Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Trainingcited by 5
  3. 2026Noise Scheduling as Information-Guided Allocation in Diffusion Trainingcited by 4
  4. 2025MMG: Mutual Information Estimation via the MMSE Gap in Diffusioncited by 3
  5. 2025Information-Theoretic Discrete Diffusioncited by 3
  6. 2026Improved Object-Centric Diffusion Learning with Registers and Contrastive Alignmentcited by 3
  7. 2026Improved Sampling Schedules for Discrete Diffusion Modelscited by 3
  8. 2026Discrete Stochastic Localization for Non-autoregressive Generationcited by 3
  9. 2025INFO-SEDD: Continuous Time Markov Chains as Scalable Information Metrics Estimatorscited by 2
  10. 2025Learning to Match Unpaired Data with Minimum Entropy Couplingcited by 2
  11. 2025Zero-Shot Image Anomaly Detection Using Generative Foundation Modelscited by 2
  12. 2025On the Separability of Information in Diffusion Modelscited by 2
  13. 2024Improving Training-free Conditional Diffusion Model via Fisher Informationcited by 1
  14. 2025TENDE: Transfer Entropy Neural Diffusion Estimationcited by 1
  15. 2026From Atoms to Entropy: Optimal Noise Allocation for Diffusion Training in the Convex Regimecited by 1
  16. 2024Conjuring Semantic Similaritycited by 0
  17. 2025Heat Diffusion Models -- Interpixel Attention Mechanismcited by 0
  18. 2025Softly Constrained Denoisers for Diffusion Models Applied to Partial Differential Equationscited by 0
  19. 2026Compensation-free Machine Unlearning in Text-to-Image Diffusion Models by Eliminating the Mutual Informationcited by 0
  20. 2026Discrete Stochastic Localization for Non-autoregressive Generationcited by 0
  21. 2026The Diffusion Encodercited by 0
  22. 2026DIPHINE: Diffusion-based Phi-ID Neural Estimatorcited by 0
  23. 2026Information-Theoretic Classifier-Free Guidance with Adaptive Schedule Optimizationcited by 0
  24. 2026Conservation Laws for Diffusion Modelscited by 0
  25. 2026PAC-DP: PAC-Bayesian Diffusion Policy Learningcited by 0
  26. 2026Information Spreading in Diffusion Models from Effective Field Theorycited by 0
  27. 2026Posterior Information Dynamics of Diffusion Models for Linear Inverse Problemscited by 0
  28. 2026Diffusion Models and Concept Formationcited by 0
  29. 2026ALICE: In-context, Zero-shot, Mutual Information Estimationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion Models: A Mathematical Introduction
  2. 2022Diffusion models as plug-and-play priors
  3. 2023Diffusion Models for Generative Artificial Intelligence: An Introduction for Applied Mathematicians
  4. 2026Feature Information Dynamics in Diffusion
  5. 2024Non-Normal Diffusion Models
  6. 2025Information Theoretic Learning for Diffusion Models with Warm Start
  7. 2025Local Diffusion Models and Phases of Data Distributions
  8. 2022Understanding Diffusion Models: A Unified Perspective