# Neural Entropic Estimation: A faster path to mutual information estimation

arXiv: https://arxiv.org/abs/1905.12957 · HTML: https://mlsanity.com/paper/1905.12957/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2019.

## Built on by (most cited first, 9 of 9)

- [Disentangling A Single MR Modality](https://mlsanity.com/paper/2205.04982/) (2022)
- [Max-Sliced Mutual Information](https://mlsanity.com/paper/2309.16200/) (2023)
- [Neural Capacity Estimators: How Reliable Are They?](https://mlsanity.com/paper/2111.07401/) (2021)
- [Graph Representation Learning via Aggregation Enhancement](https://mlsanity.com/paper/2201.12843/) (2022)
- [A Perspective on Neural Capacity Estimation: Viability and Reliability](https://mlsanity.com/paper/2203.11793/) (2022)
- [k-Sliced Mutual Information: A Quantitative Study of Scalability with Dimension](https://mlsanity.com/paper/2206.08526/) (2022)
- [Lower Bounds on the MMSE of Adversarially Inferring Sensitive Features](https://mlsanity.com/paper/2505.09004/) (2025)
- [A Neural Network Algorithm for KL Divergence Estimation with Quantitative Error Bounds](https://mlsanity.com/paper/2510.05386/) (2025)
- [CLaST: Context-aware Contrastive VAE for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2608.20025/) (2026)

## Abstract (arXiv)

We point out a limitation of the mutual information neural estimation (MINE) where the network fails to learn at the initial training phase, leading to slow convergence in the number of training iterations. To solve this problem, we propose a faster method called the mutual information neural entropic estimation (MI-NEE). Our solution first generalizes MINE to estimate the entropy using a custom reference distribution. The entropy estimate can then be used to estimate the mutual information. We argue that the seemingly redundant intermediate step of entropy estimation allows one to improve the convergence by an appropriate reference distribution. In particular, we show that MI-NEE reduces to MINE in the special case when the reference distribution is the product of marginal distributions, but faster convergence is possible by choosing the uniform distribution as the reference distribution instead. Compared to the product of marginals, the uniform distribution introduces more samples in low-density regions and fewer samples in high-density regions, which appear to lead to an overall larger gradient for faster convergence.