# Accurate Estimation of Mutual Information in High Dimensional Data

arXiv: https://arxiv.org/abs/2506.00330 · HTML: https://mlsanity.com/paper/2506.00330/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 75 references that are in the map.
- First posted on arXiv 31 May 2025.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [On Deep Multi-View Representation Learning: Objectives and Optimization](https://mlsanity.com/paper/1602.01024/) (2016)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [Efficient Estimation of Mutual Information for Strongly Dependent Variables](https://mlsanity.com/paper/1411.2003/) (2014)
- [Deep Variational Canonical Correlation Analysis](https://mlsanity.com/paper/1610.03454/) (2016)
- [Beyond Normal: On the Evaluation of Mutual Information Estimators](https://mlsanity.com/paper/2306.11078/) (2023)
- [MINDE: Mutual Information Neural Diffusion Estimation](https://mlsanity.com/paper/2310.09031/) (2023)
- [Predictive information in a sensory population](https://mlsanity.com/paper/1307.0225/) (2013)
- [Mutual Information Estimation via Normalizing Flows](https://mlsanity.com/paper/2403.02187/) (2024)
- [Approximating mutual information of high-dimensional variables using learned representations](https://mlsanity.com/paper/2409.02732/) (2024)
- [Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization](https://mlsanity.com/paper/2107.01131/) (2021)
- [Multivariate Information Bottleneck](https://mlsanity.com/paper/1301.2270/) (2013)
- [Mutual Information Estimation via f-Divergence and Data Derangements](https://mlsanity.com/paper/2305.20025/) (2023)
- [Generative power of a protein language model trained on multiple sequence alignments](https://mlsanity.com/paper/2204.07110/) (2022)
- [Data-Efficient Mutual Information Neural Estimator](https://mlsanity.com/paper/1905.03319/) (2019)
- [Deep Variational Multivariate Information Bottleneck -- A Framework for Variational Losses](https://mlsanity.com/paper/2310.03311/) (2023)
- [Simultaneous Dimensionality Reduction: A Data Efficient Approach for Multimodal Representations Learning](https://mlsanity.com/paper/2310.04458/) (2023)
- [Quantifying how much sensory information in a neural code is relevant for behavior](https://mlsanity.com/paper/1712.02449/) (2017)
- [Millisecond-scale motor encoding in a cortical vocal area](https://mlsanity.com/paper/1404.0669/) (2014)

## Built on by (most cited first, 2 of 2)

- [Mutual information and task-relevant latent dimensionality](https://mlsanity.com/paper/2602.08105/) (2026)
- [Information bottleneck for learning the phase space of dynamics from high-dimensional experimental data](https://mlsanity.com/paper/2604.24662/) (2026)

## Abstract (arXiv)

Mutual information (MI) quantifies statistical dependence between variables and is widely used across scientific disciplines, yet accurate estimation from finite data remains notoriously difficult. Common approaches fail in high-dimensional, undersampled regimes (N lesssim K) typical of modern experiments, and no accepted tests exist to detect when neural network-based estimators fail, making them effectively unusable as scientific instruments. We show that neural MI estimators can be made reliable when the statistical dependencies admit a low-dimensional latent representation. Sample complexity is then governed by the latent dimensionality KZ ll K rather than the ambient dimension -- a regime shift we confirm empirically and ground theoretically via random matrix theory. Building on this insight, we develop a practical protocol that provides neural estimators with explicit statistical consistency checks, bias correction, and confidence intervals. We additionally introduce a new class of probabilistic critics (the VSIB family) that substantially reduce bias and variance at higher MI values where standard estimators break down. We validate the protocol on synthetic benchmarks (K=500, N as low as 256), on the standard 40-dataset benchmark suite of Czyz et al. (2023), on noisy MNIST (K=784), and on CIFAR-10/100 (K=3072) with a ResNet-20 backbone. Our protocol consistently matches or exceeds existing methods while being the only approach to report confidence intervals and flag unreliable estimates, achieving reliable MI detection well below the ambient pixel dimension on real images.