# DEMI: Discriminative Estimator of Mutual Information

arXiv: https://arxiv.org/abs/2010.01766 · HTML: https://mlsanity.com/paper/2010.01766/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2020.

## Built on by (most cited first, 5 of 5)

- [Multimodal Representation Learning via Maximization of Local Mutual Information](https://mlsanity.com/paper/2103.04537/) (2021)
- [Boosting Semi-supervised Image Segmentation with Global and Local Mutual Information Regularization](https://mlsanity.com/paper/2103.04813/) (2021)
- [Connecting Jensen-Shannon and Kullback-Leibler Divergences: A New Bound for Representation Learning](https://mlsanity.com/paper/2510.20644/) (2025)
- [Neural Mutual Information Estimation with Vector Copulas](https://mlsanity.com/paper/2510.20968/) (2025)
- [A Neural Difference-of-Entropies Estimator for Mutual Information](https://mlsanity.com/paper/2502.13085/) (2025)

## Abstract (arXiv)

Estimating mutual information between continuous random variables is often intractable and extremely challenging for high-dimensional data. Recent progress has leveraged neural networks to optimize variational lower bounds on mutual information. Although showing promise for this difficult problem, the variational methods have been theoretically and empirically proven to have serious statistical limitations: 1) many methods struggle to produce accurate estimates when the underlying mutual information is either low or high; 2) the resulting estimators may suffer from high variance. Our approach is based on training a classifier that provides the probability that a data sample pair is drawn from the joint distribution rather than from the product of its marginal distributions. Moreover, we establish a direct connection between mutual information and the average log odds estimate produced by the classifier on a test set, leading to a simple and accurate estimator of mutual information. We show theoretically that our method and other variational approaches are equivalent when they achieve their optimum, while our method sidesteps the variational bound. Empirical results demonstrate high accuracy of our approach and the advantages of our estimator in the context of representation learning. Our demo is available at https://github.com/RayRuizhiLiao/demi_mi_estimator.