# MINDE: Mutual Information Neural Diffusion Estimation

arXiv: https://arxiv.org/abs/2310.09031 · HTML: https://mlsanity.com/paper/2310.09031/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 59 references that are in the map.
- First posted on arXiv 13 Oct 2023.

## Builds on (most cited first, 30 of 41)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions](https://mlsanity.com/paper/2209.11215/) (2022)
- [Convergence of denoising diffusion models under the manifold hypothesis](https://mlsanity.com/paper/2208.05314/) (2022)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale](https://mlsanity.com/paper/2303.06555/) (2023)
- [A Variational Perspective on Diffusion-Based Generative Models and Score Matching](https://mlsanity.com/paper/2106.02808/) (2021)
- [Convergence for score-based generative modeling with polynomial complexity](https://mlsanity.com/paper/2206.06227/) (2022)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [Telescoping Density-Ratio Estimation](https://mlsanity.com/paper/2006.12204/) (2020)
- [On Nesting Monte Carlo Estimators](https://mlsanity.com/paper/1709.06181/) (2017)
- [Efficient Estimation of Mutual Information for Strongly Dependent Variables](https://mlsanity.com/paper/1411.2003/) (2014)
- [Information-Theoretic Diffusion](https://mlsanity.com/paper/2302.03792/) (2023)
- [Beyond Normal: On the Evaluation of Mutual Information Estimators](https://mlsanity.com/paper/2306.11078/) (2023)
- [How Much is Enough? A Study on Diffusion Times in Score-based Generative Models](https://mlsanity.com/paper/2206.05173/) (2022)

## Built on by (most cited first, 30 of 31)

- [Mutual Information Estimation via Normalizing Flows](https://mlsanity.com/paper/2403.02187/) (2024)
- [Information Theoretic Text-to-Image Alignment](https://mlsanity.com/paper/2405.20759/) (2024)
- [Neural Entropy](https://mlsanity.com/paper/2409.03817/) (2024)
- [InfoBridge: Mutual Information estimation via Bridge Matching](https://mlsanity.com/paper/2502.01383/) (2025)
- [Noise Scheduling as Information-Guided Allocation in Diffusion Training](https://mlsanity.com/paper/2602.18647/) (2026)
- [Mutual Information Preserving Neural Network Pruning](https://mlsanity.com/paper/2411.00147/) (2024)
- [Latent Abstractions in Generative Diffusion Models](https://mlsanity.com/paper/2410.03368/) (2024)
- [Neural Mutual Information Estimation with Vector Copulas](https://mlsanity.com/paper/2510.20968/) (2025)
- [SOmegaI: Score-based O-INFORMATION Estimation](https://mlsanity.com/paper/2402.05667/) (2024)
- [MMG: Mutual Information Estimation via the MMSE Gap in Diffusion](https://mlsanity.com/paper/2509.20609/) (2025)
- [Improved Sampling Schedules for Discrete Diffusion Models](https://mlsanity.com/paper/2602.06849/) (2026)
- [INFO-SEDD: Continuous Time Markov Chains as Scalable Information Metrics Estimators](https://mlsanity.com/paper/2502.19183/) (2025)
- [Accurate Estimation of Mutual Information in High Dimensional Data](https://mlsanity.com/paper/2506.00330/) (2025)
- [Why Do Unlearnable Examples Work: A Novel Perspective of Mutual Information](https://mlsanity.com/paper/2603.03725/) (2026)
- [A Neural Difference-of-Entropies Estimator for Mutual Information](https://mlsanity.com/paper/2502.13085/) (2025)
- [TENDE: Transfer Entropy Neural Diffusion Estimation](https://mlsanity.com/paper/2510.14096/) (2025)
- [MIST: Mutual Information Estimation Via Supervised Training](https://mlsanity.com/paper/2511.18945/) (2025)
- [Discrete Bridges for Mutual Information Estimation](https://mlsanity.com/paper/2602.08894/) (2026)
- [Amortized Vine Copulas for High-Dimensional Density and Information Estimation](https://mlsanity.com/paper/2604.20568/) (2026)
- [Relative Entropy Estimation in Function Space: Theory and Applications to Trajectory Inference](https://mlsanity.com/paper/2604.20775/) (2026)
- [InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate](https://mlsanity.com/paper/2606.00241/) (2026)
- [From Atoms to Entropy: Optimal Noise Allocation for Diffusion Training in the Convex Regime](https://mlsanity.com/paper/2607.20540/) (2026)
- [Mutual Information Multinomial Estimation](https://mlsanity.com/paper/2408.09377/) (2024)
- [Disentangled representations via score-based variational autoencoders](https://mlsanity.com/paper/2512.17127/) (2025)
- [GeoIB: Geometry-Aware Information Bottleneck via Statistical-Manifold Compression](https://mlsanity.com/paper/2602.03906/) (2026)
- [The Diffusion Encoder](https://mlsanity.com/paper/2605.13399/) (2026)
- [DIPHINE: Diffusion-based Phi-ID Neural Estimator](https://mlsanity.com/paper/2606.18997/) (2026)
- [Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation](https://mlsanity.com/paper/2607.03487/) (2026)
- [Posterior Information Dynamics of Diffusion Models for Linear Inverse Problems](https://mlsanity.com/paper/2608.21709/) (2026)
- [ALICE: In-context, Zero-shot, Mutual Information Estimation](https://mlsanity.com/paper/2609.34962/) (2026)

## Abstract (arXiv)

In this work we present a new method for the estimation of Mutual Information (MI) between random variables. Our approach is based on an original interpretation of the Girsanov theorem, which allows us to use score-based diffusion models to estimate the Kullback Leibler divergence between two densities as a difference between their score functions. As a by-product, our method also enables the estimation of the entropy of random variables. Armed with such building blocks, we present a general recipe to measure MI, which unfolds in two directions: one uses conditional diffusion process, whereas the other uses joint diffusion processes that allow simultaneous modelling of two random variables. Our results, which derive from a thorough experimental protocol over all the variants of our approach, indicate that our method is more accurate than the main alternatives from the literature, especially for challenging distributions. Furthermore, our methods pass MI self-consistency tests, including data processing and additivity under independence, which instead are a pain-point of existing methods.