mlsanityData to 7 Oct 2026

MINDE: Mutual Information Neural Diffusion Estimation

arXiv 2310.09031 ↗First posted 13 Oct 2023cs.LG stat.MLcited by 31

Not on the map. Why?

Abstract (arXiv)

In this work we present a new method for the estimation of Mutual Information (MI) between random variables. Our approach is based on an original interpretation of the Girsanov theorem, which allows us to use score-based diffusion models to estimate the Kullback Leibler divergence between two densities as a difference between their score functions. As a by-product, our method also enables the estimation of the entropy of random variables. Armed with such building blocks, we present a general recipe to measure MI, which unfolds in two directions: one uses conditional diffusion process, whereas the other uses joint diffusion processes that allow simultaneous modelling of two random variables. Our results, which derive from a thorough experimental protocol over all the variants of our approach, indicate that our method is more accurate than the main alternatives from the literature, especially for challenging distributions. Furthermore, our methods pass MI self-consistency tests, including data processing and additivity under independence, which instead are a pain-point of existing methods.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 41 of its 59 references that are in the map. First posted on arXiv 13 Oct 2023.

Builds on

41 of 59 references.

References · 41

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  10. 2021Variational Diffusion Modelscited by 1,145
  11. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  12. 2016Deep Variational Information Bottleneckcited by 915
29 more
  1. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  2. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  3. 2018MINE: Mutual Information Neural Estimationcited by 680
  4. 2019On Variational Bounds of Mutual Informationcited by 550
  5. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  6. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptionscited by 440
  7. 2022Convergence of denoising diffusion models under the manifold hypothesiscited by 237
  8. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  9. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
  10. 2021A Variational Perspective on Diffusion-Based Generative Models and Score Matchingcited by 182
  11. 2022Convergence for score-based generative modeling with polynomial complexitycited by 176
  12. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  13. 2020Telescoping Density-Ratio Estimationcited by 108
  14. 2017On Nesting Monte Carlo Estimatorscited by 82
  15. 2014Efficient Estimation of Mutual Information for Strongly Dependent Variablescited by 46
  16. 2023Information-Theoretic Diffusioncited by 41
  17. 2023Beyond Normal: On the Evaluation of Mutual Information Estimatorscited by 37
  18. 2022How Much is Enough? A Study on Diffusion Times in Score-based Generative Modelscited by 35
  19. 2022Diagnosing and Fixing Manifold Overfitting in Deep Generative Modelscited by 28
  20. 2018Mutual Information Maximization for Simple and Accurate Part-Of-Speech Inductioncited by 17
  21. 2023Mutual Information Estimation via f-Divergence and Data Derangementscited by 17
  22. 2020Evaluating Lossy Compression Rates of Deep Generative Modelscited by 16
  23. 2022Copula Density Neural Estimationcited by 13
  24. 2018GILBO: One Metric to Measure Them Allcited by 9
  25. 2023Multi-modal Latent Diffusioncited by 9
  26. 2023Improving Mutual Information Estimation with Annealed and Energy-Based Boundscited by 8
  27. 2023On the Effectiveness of Hybrid Mutual Information Estimationcited by 6
  28. 2018The Information Autoencoding Family: A Lagrangian Perspective on Latent Variable Generative Modelscited by 5
  29. 2021A Reverse Jensen Inequality Result with Application to Mutual Information Estimationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 9; 2026: 15.

31 papers.

Sort:

By area:

  1. 2024Mutual Information Estimation via Normalizing Flowscited by 22
  2. 2024Information Theoretic Text-to-Image Alignmentcited by 8
  3. 2024Neural Entropycited by 7
  4. 2025InfoBridge: Mutual Information estimation via Bridge Matchingcited by 7
  5. 2024Mutual Information Preserving Neural Network Pruningcited by 5
  6. 2024Latent Abstractions in Generative Diffusion Modelscited by 4
  7. 2025Neural Mutual Information Estimation with Vector Copulascited by 4
  8. 2026Noise Scheduling as Information-Guided Allocation in Diffusion Trainingcited by 4
  9. 2024SOmegaI: Score-based O-INFORMATION Estimationcited by 3
  10. 2025MMG: Mutual Information Estimation via the MMSE Gap in Diffusioncited by 3
  11. 2026Improved Sampling Schedules for Discrete Diffusion Modelscited by 3
  12. 2025INFO-SEDD: Continuous Time Markov Chains as Scalable Information Metrics Estimatorscited by 2
19 more
  1. 2025Accurate Estimation of Mutual Information in High Dimensional Datacited by 2
  2. 2026Why Do Unlearnable Examples Work: A Novel Perspective of Mutual Informationcited by 2
  3. 2025A Neural Difference-of-Entropies Estimator for Mutual Informationcited by 1
  4. 2025TENDE: Transfer Entropy Neural Diffusion Estimationcited by 1
  5. 2025MIST: Mutual Information Estimation Via Supervised Trainingcited by 1
  6. 2026Discrete Bridges for Mutual Information Estimationcited by 1
  7. 2026Amortized Vine Copulas for High-Dimensional Density and Information Estimationcited by 1
  8. 2026Relative Entropy Estimation in Function Space: Theory and Applications to Trajectory Inferencecited by 1
  9. 2026InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimatecited by 1
  10. 2026From Atoms to Entropy: Optimal Noise Allocation for Diffusion Training in the Convex Regimecited by 1
  11. 2024Mutual Information Multinomial Estimationcited by 0
  12. 2025Disentangled representations via score-based variational autoencoderscited by 0
  13. 2026GeoIB: Geometry-Aware Information Bottleneck via Statistical-Manifold Compressioncited by 0
  14. 2026The Diffusion Encodercited by 0
  15. 2026DIPHINE: Diffusion-based Phi-ID Neural Estimatorcited by 0
  16. 2026Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimationcited by 0
  17. 2026Posterior Information Dynamics of Diffusion Models for Linear Inverse Problemscited by 0
  18. 2026ALICE: In-context, Zero-shot, Mutual Information Estimationcited by 0
  19. 2026Controlling Dependence in Implicit Generative Models via Spread Mutual Informationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Information-Theoretic Discrete Diffusion
  2. 2026An exact information theory of generalization phase transitions in Bayesian diffusion models
  3. 2026Rethinking the Diffusion Model from a Langevin Perspective
  4. 2024The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications
  5. 2026From Score Approximation to Distribution Approximation in Score-Based Diffusion Models
  6. 2026First Learn, Then Memorize: The Spectral Bias of Diffusion Models
  7. 2026Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
  8. 2025On the Separability of Information in Diffusion Models