Conservation Laws for Diffusion Models
On the map: Diffusion models → Diffusion foundations and sampling
Abstract (arXiv)
While autoregressive models optimize the exact data likelihood via the chain rule, diffusion models are typically trained with denoising objectives. We develop conservation laws based on generalized extrinsic information transfer (GEXIT) functions for a broad class of memoryless noise processes, showing that the data--model cross-entropy (CE) can be characterized exactly as an integral of local information-theoretic derivatives along the noise path. This yields a unified characterization of the likelihood for discrete and continuous diffusion, with the Gaussian case reducing to the well-known mutual information--minimum mean-square error (I-MMSE) relationship. An immediate implication is a locality property: one can compute the information-theoretic derivatives using only the marginal posteriors along the noise path. As a result, training reduces to learning the marginal posteriors by minimizing the negative log-likelihood. While the conservation law implies that the entropy does not depend on the noise path, finite-capacity denoisers approximate the posteriors with varying accuracy across noise types, leading to differences in performance. We validate these predictions on synthetic Markov sources and standard benchmarks, including text8 and CIFAR-10.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 21 of its 33 references that are in the map. First posted on arXiv 11 Jul 2026.
Builds on
21 of 33 references.
Closest ancestors · 11
Same direction, newest first
- 2025Information-Theoretic Discrete Diffusion28 Oct
- 2025ItDPDM: Information-Theoretic Discrete Poisson Diffusion Model8 May
- 2025Information-Theoretic Proofs for Diffusion Sampling4 Feb
- 2024Improving Vector-Quantized Image Modeling with Latent Consistency-Matching Diffusion18 Oct
- 2023Information-Theoretic Diffusion7 Feb
- 2022Scalable Diffusion Models with Transformers19 Dec
- 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul
- 2021Improved Denoising Diffusion Probabilistic Models18 Feb
- 2021Maximum Likelihood Training of Score-Based Diffusion Models22 Jan
- 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
- 2020Denoising Diffusion Probabilistic Models19 Jun
Other references · 10
- 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
- 2016Improved Techniques for Training GANscited by 2,970
- 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
- 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
- 2024Simple and Effective Masked Diffusion Language Modelscited by 848
- 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
- 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
- 2023Likelihood-Based Diffusion Language Modelscited by 151
- 2024Energy-Based Diffusion Language Models for Text Generationcited by 89
- 2022Empowering Diffusion Models on the Embedding Space for Text Generationcited by 40
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation
- 2025Local Diffusion Models and Phases of Data Distributions
- 2025Information Theoretic Learning for Diffusion Models with Warm Start
- 2025Assessing the Quality of Denoising Diffusion Models in Wasserstein Distance: Noisy Score and Optimal Bounds
- 2025Diffusion Models: A Mathematical Introduction
- 2025Diffusion models under low-noise regime
- 2026A theory of learning data statistics in diffusion models, from easy to hard
- 2025Diffusion Models under Alternative Noise: Simplified Analysis and Sensitivity