# A Neural Difference-of-Entropies Estimator for Mutual Information

arXiv: https://arxiv.org/abs/2502.13085 · HTML: https://mlsanity.com/paper/2502.13085/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Builds on (most cited first, 20 of 20)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Learning to Explain: An Information-Theoretic Perspective on Model Interpretation](https://mlsanity.com/paper/1802.07814/) (2018)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [A Review of Feature Selection Methods Based on Mutual Information](https://mlsanity.com/paper/1509.07577/) (2015)
- [Block Neural Autoregressive Flow](https://mlsanity.com/paper/1904.04676/) (2019)
- [CCMI : Classifier based Conditional Mutual Information Estimation](https://mlsanity.com/paper/1906.01824/) (2019)
- [Beyond Normal: On the Evaluation of Mutual Information Estimators](https://mlsanity.com/paper/2306.11078/) (2023)
- [Interpretable Diffusion via Information Decomposition](https://mlsanity.com/paper/2310.07972/) (2023)
- [MINDE: Mutual Information Neural Diffusion Estimation](https://mlsanity.com/paper/2310.09031/) (2023)
- [Mutual Information Estimation via Normalizing Flows](https://mlsanity.com/paper/2403.02187/) (2024)
- [Demystifying Fixed k-Nearest Neighbor Information Estimators](https://mlsanity.com/paper/1604.03006/) (2016)
- [Diffeomorphic Information Neural Estimation](https://mlsanity.com/paper/2211.10856/) (2022)
- [A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets](https://mlsanity.com/paper/2410.10924/) (2024)
- [DEMI: Discriminative Estimator of Mutual Information](https://mlsanity.com/paper/2010.01766/) (2020)
- [Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?](https://mlsanity.com/paper/2409.16697/) (2024)

## Built on by (most cited first, 1 of 1)

- [Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation](https://mlsanity.com/paper/2607.03487/) (2026)

## Abstract (arXiv)

Estimating Mutual Information (MI), a key measure of dependence of random quantities without specific modelling assumptions, is a challenging problem in high dimensions. We propose a novel mutual information estimator based on parametrizing conditional densities using normalizing flows, a deep generative model that has gained popularity in recent years. This estimator leverages a block autoregressive structure to achieve improved bias-variance trade-offs on standard benchmark tasks.