# Data-Efficient Mutual Information Neural Estimator

arXiv: https://arxiv.org/abs/1905.03319 · HTML: https://mlsanity.com/paper/1905.03319/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2019.

## Built on by (most cited first, 6 of 6)

- [Information-Theoretic Bias Reduction via Causal View of Spurious Correlation](https://mlsanity.com/paper/2201.03121/) (2022)
- [First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization](https://mlsanity.com/paper/2205.12381/) (2022)
- [Accurate Estimation of Mutual Information in High Dimensional Data](https://mlsanity.com/paper/2506.00330/) (2025)
- [FedPall: Prototype-based Adversarial and Collaborative Learning for Federated Learning with Feature Drift](https://mlsanity.com/paper/2507.04781/) (2025)
- [Improving Numerical Stability of Normalized Mutual Information Estimator on High Dimensions](https://mlsanity.com/paper/2410.07642/) (2024)
- [A Neural Network Algorithm for KL Divergence Estimation with Quantitative Error Bounds](https://mlsanity.com/paper/2510.05386/) (2025)

## Abstract (arXiv)

Measuring Mutual Information (MI) between high-dimensional, continuous, random variables from observed samples has wide theoretical and practical applications. Recent work, MINE (Belghazi et al. 2018), focused on estimating tight variational lower bounds of MI using neural networks, but assumed unlimited supply of samples to prevent overfitting. In real world applications, data is not always available at a surplus. In this work, we focus on improving data efficiency and propose a Data-Efficient MINE Estimator (DEMINE), by developing a relaxed predictive MI lower bound that can be estimated at higher data efficiency by orders of magnitudes. The predictive MI lower bound also enables us to develop a new meta-learning approach using task augmentation, Meta-DEMINE, to improve generalization of the network and further boost estimation accuracy empirically. With improved data-efficiency, our estimators enables statistical testing of dependency at practical dataset sizes. We demonstrate the effectiveness of our estimators on synthetic benchmarks and a real world fMRI data, with application of inter-subject correlation analysis.