mlsanityData to 14 Oct 2026

Neural Entropy

arXiv 2409.03817 ↗First posted 5 Sep 2024cs.LG cond-mat.stat-mech cs.ITcited by 7

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

We explore the connection between deep learning and information theory through the paradigm of diffusion models. A diffusion model converts noise into structured data by reinstating, imperfectly, information that is erased when data was diffused to noise. This information is stored in a neural network during training. We quantify this information by introducing a measure called neural entropy, which is related to the total entropy produced by diffusion. Neural entropy is a function of not just the data distribution, but also the diffusive process itself. Measurements of neural entropy on a few simple image diffusion models reveal that they are extremely efficient at compressing large ensembles of structured data.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 30 of its 92 references that are in the map. First posted on arXiv 5 Sep 2024.

Builds on

30 of 92 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023Generative Diffusion From An Action Principle6 Oct
  2. 2023Generalization in diffusion models arises from geometry-adaptive harmonic representations4 Oct
  3. 2023Diffusion Schr"odinger Bridge Matching29 Mar
  4. 2023Information-Theoretic Diffusion7 Feb
  5. 2023On the Mathematics of Diffusion Models25 Jan
  6. 2022Scalable Diffusion Models with Transformers19 Dec
  7. 2022Score-based Generative Modeling Secretly Minimizes the Wasserstein Distance13 Dec
  8. 2022An optimal control perspective on diffusion-based generative modeling2 Nov
  9. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions22 Sep
  10. 2021Score-Based Generative Modeling with Critically-Damped Langevin Diffusion14 Dec
  11. 2021Variational Diffusion Models1 Jul
  12. 2021Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation10 Jun
4 more
  1. 2021A Variational Perspective on Diffusion-Based Generative Models and Score Matching5 Jun
  2. 2021Maximum Likelihood Training of Score-Based Diffusion Models22 Jan
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  4. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 14

  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2022Flow Matching for Generative Modelingcited by 5,702
  4. 2017Non-local Neural Networkscited by 2,293
  5. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  6. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  7. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  8. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  9. 2021Path Integral Sampler: a stochastic control approach for samplingcited by 181
  10. 2020Interacting particle solutions of Fokker-Planck equations through gradient-log-density estimationcited by 88
  11. 2024Optimal Flow Matching: Learning Straight Trajectories in Just One Stepcited by 78
  12. 2023MINDE: Mutual Information Neural Diffusion Estimationcited by 31
2 more
  1. 2024Lossless data compression by large modelscited by 20
  2. 2023Bayesian Renormalizationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025Entropic Time Schedulers for Generative Diffusion Modelscited by 16
  2. 2025The Information Dynamics of Generative Diffusioncited by 13
  3. 2026The Entropic Signature of Class Speciation in Diffusion Modelscited by 9
  4. 2026Noise Scheduling as Information-Guided Allocation in Diffusion Trainingcited by 7
  5. 2026Improved Sampling Schedules for Discrete Diffusion Modelscited by 3
  6. 2026The Diffusion Encodercited by 0
  7. 2026A Unified Kullback--Leibler Divergence Analysis of Generative Diffusion Models via Entropy Production Ratecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On the Separability of Information in Diffusion Models
  2. 2026A theory of learning data statistics in diffusion models, from easy to hard
  3. 2026From Atoms to Entropy: Optimal Noise Allocation for Diffusion Training in the Convex Regime
  4. 2026When Diffusion Model Can Ignore Dimension: An Entropy-Based Theory
  5. 2026An exact information theory of generalization phase transitions in Bayesian diffusion models
  6. 2025Measuring Semantic Information Production in Generative Diffusion Models
  7. 2024Transformers from Diffusion: A Unified Framework for Neural Message Passing
  8. 2026Feature Information Dynamics in Diffusion