# Information-Theoretic Diffusion

arXiv: https://arxiv.org/abs/2302.03792 · HTML: https://mlsanity.com/paper/2302.03792/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 24 of its 49 references that are in the map.
- First posted on arXiv 7 Feb 2023.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 24 of 24)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning](https://mlsanity.com/paper/2208.04202/) (2022)
- [Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise](https://mlsanity.com/paper/2208.09392/) (2022)
- [Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design](https://mlsanity.com/paper/1902.00275/) (2019)
- [A Variational Perspective on Diffusion-Based Generative Models and Score Matching](https://mlsanity.com/paper/2106.02808/) (2021)
- [Let us Build Bridges: Understanding and Extending Diffusion Generative Models](https://mlsanity.com/paper/2208.14699/) (2022)
- [The Thermodynamic Variational Objective](https://mlsanity.com/paper/1907.00031/) (2019)
- [q-Paths: Generalizing the Geometric Annealing Path using Power Means](https://mlsanity.com/paper/2107.00745/) (2021)
- [All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference](https://mlsanity.com/paper/2007.00642/) (2020)
- [Variational Inference with Holder Bounds](https://mlsanity.com/paper/2111.02947/) (2021)
- [Generalized Bregman Divergence and Gradient of Mutual Information for Vector Poisson Channels](https://mlsanity.com/paper/1301.6648/) (2013)

## Built on by (most cited first, 30 of 44)

- [Interpretable Diffusion via Information Decomposition](https://mlsanity.com/paper/2310.07972/) (2023)
- [Maximum Entropy Reinforcement Learning with Diffusion Policy](https://mlsanity.com/paper/2502.11612/) (2025)
- [Esoteric Language Models: A Family of Any-Order Diffusion LLMs](https://mlsanity.com/paper/2506.01928/) (2025)
- [MINDE: Mutual Information Neural Diffusion Estimation](https://mlsanity.com/paper/2310.09031/) (2023)
- [CPR: Retrieval Augmented Generation for Copyright Protection](https://mlsanity.com/paper/2403.18920/) (2024)
- [Learning normalized image densities via dual score matching](https://mlsanity.com/paper/2506.05310/) (2025)
- [Training Data Protection with Compositional Diffusion Models](https://mlsanity.com/paper/2308.01937/) (2023)
- [Entropic Time Schedulers for Generative Diffusion Models](https://mlsanity.com/paper/2504.13612/) (2025)
- [Understanding Diffusion Models by Feynman's Path Integral](https://mlsanity.com/paper/2403.11262/) (2024)
- [The Information Dynamics of Generative Diffusion](https://mlsanity.com/paper/2508.19897/) (2025)
- [The Entropic Signature of Class Speciation in Diffusion Models](https://mlsanity.com/paper/2602.09651/) (2026)
- [Neural Entropy](https://mlsanity.com/paper/2409.03817/) (2024)
- [Noise Scheduling as Information-Guided Allocation in Diffusion Training](https://mlsanity.com/paper/2602.18647/) (2026)
- [Information-Theoretic Proofs for Diffusion Sampling](https://mlsanity.com/paper/2502.02305/) (2025)
- [Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training](https://mlsanity.com/paper/2407.08946/) (2024)
- [MMG: Mutual Information Estimation via the MMSE Gap in Diffusion](https://mlsanity.com/paper/2509.20609/) (2025)
- [Information-Theoretic Discrete Diffusion](https://mlsanity.com/paper/2510.24088/) (2025)
- [Improved Object-Centric Diffusion Learning with Registers and Contrastive Alignment](https://mlsanity.com/paper/2601.01224/) (2026)
- [Improved Sampling Schedules for Discrete Diffusion Models](https://mlsanity.com/paper/2602.06849/) (2026)
- [Discrete Stochastic Localization for Non-autoregressive Generation](https://mlsanity.com/paper/2602.16169/) (2026)
- [INFO-SEDD: Continuous Time Markov Chains as Scalable Information Metrics Estimators](https://mlsanity.com/paper/2502.19183/) (2025)
- [Learning to Match Unpaired Data with Minimum Entropy Coupling](https://mlsanity.com/paper/2503.08501/) (2025)
- [Zero-Shot Image Anomaly Detection Using Generative Foundation Models](https://mlsanity.com/paper/2507.22692/) (2025)
- [On the Separability of Information in Diffusion Models](https://mlsanity.com/paper/2509.23937/) (2025)
- [Improving Training-free Conditional Diffusion Model via Fisher Information](https://mlsanity.com/paper/2404.18252/) (2024)
- [TENDE: Transfer Entropy Neural Diffusion Estimation](https://mlsanity.com/paper/2510.14096/) (2025)
- [Softly Constrained Denoisers for Diffusion Models Applied to Partial Differential Equations](https://mlsanity.com/paper/2512.14980/) (2025)
- [From Atoms to Entropy: Optimal Noise Allocation for Diffusion Training in the Convex Regime](https://mlsanity.com/paper/2607.20540/) (2026)
- [Two-Sample Testing via Path-based Inference](https://mlsanity.com/paper/2610.05684/) (2026)
- [Conjuring Semantic Similarity](https://mlsanity.com/paper/2410.16431/) (2024)

## Abstract (arXiv)

Denoising diffusion models have spurred significant gains in density modeling and image generation, precipitating an industrial revolution in text-guided AI art generation. We introduce a new mathematical foundation for diffusion models inspired by classic results in information theory that connect Information with Minimum Mean Square Error regression, the so-called I-MMSE relations. We generalize the I-MMSE relations to exactly relate the data distribution to an optimal denoising regression problem, leading to an elegant refinement of existing diffusion bounds. This new insight leads to several improvements for probability distribution estimation, including theoretical justification for diffusion model ensembling. Remarkably, our framework shows how continuous and discrete probabilities can be learned with the same regression objective, avoiding domain-specific generative models used in variational methods. Code to reproduce experiments is provided at http://github.com/kxh001/ITdiffusion and simplified demonstration code is at http://github.com/gregversteeg/InfoDiffusionSimple.