mlsanityData to 14 Oct 2026

Beyond Normal: On the Evaluation of Mutual Information Estimators

arXiv 2306.11078 ↗First posted 19 Jun 2023stat.ML cs.IT cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

Mutual information is a general statistical dependency measure which has found applications in representation learning, causality, domain generalization and computational biology. However, mutual information estimators are typically evaluated on simple families of probability distributions, namely multivariate normal distribution and selected distributions with one-dimensional random variables. In this paper, we show how to construct a diverse family of distributions with known ground-truth mutual information and propose a language-independent benchmarking platform for mutual information estimators. We discuss the general applicability and limitations of classical and neural estimators in settings involving high dimensions, sparse interactions, long-tailed distributions, and high mutual information. Finally, we provide guidelines for practitioners on how to select appropriate estimator adapted to the difficulty of problem considered and issues one needs to consider when applying an estimator to a new data set.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 22 of its 81 references that are in the map. First posted on arXiv 19 Jun 2023.

Builds on

22 of 81 references.

References · 22

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2018MINE: Mutual Information Neural Estimationcited by 683
  3. 2019On Variational Bounds of Mutual Informationcited by 559
  4. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
  5. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 503
  6. 2021Deep Long-Tailed Learning: A Surveycited by 417
  7. 2019On Mutual Information Maximization for Representation Learningcited by 316
  8. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
  9. 2021Contrastive Learning Inverts the Data Generating Processcited by 221
  10. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  11. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  12. 2021Invariant Information Bottleneck for Domain Generalizationcited by 86
10 more
  1. 2023Identifiability Results for Multimodal Contrastive Learningcited by 61
  2. 2018Learning Plannable Representations with Causal InfoGANcited by 57
  3. 2014Efficient Estimation of Mutual Information for Strongly Dependent Variablescited by 46
  4. 2017Estimating Mutual Information for Discrete-Continuous Mixturescited by 34
  5. 2020Learning Unbiased Representations via Mutual Information Backpropagationcited by 25
  6. 2018Only Bayes should learn a manifold (on the estimation of differential geometric structure from data)cited by 17
  7. 2020Statistical Consequences of Fat Tails: Real World Preasymptotics, Epistemology, and Applicationscited by 16
  8. 2016Demystifying Fixed k-Nearest Neighbor Information Estimatorscited by 14
  9. 2019Testing Conditional Independence on Discrete Data using Stochastic Complexitycited by 10
  10. 2017Geometric k-nearest neighbor estimation of entropy and mutual informationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 12; 2025: 15; 2026: 7.

37 papers.

Sort:

By area:

  1. 2023MINDE: Mutual Information Neural Diffusion Estimationcited by 31
  2. 2024Mutual Information Estimation via Normalizing Flowscited by 22
  3. 2024Approximating mutual information of high-dimensional variables using learned representationscited by 21
  4. 2025Leakage and Interpretability in Concept-Based Modelscited by 13
  5. 2024A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasetscited by 10
  6. 2023On the Properties and Estimation of Pointwise Mutual Information Profilescited by 7
  7. 2025InfoBridge: Mutual Information estimation via Bridge Matchingcited by 7
  8. 202410 Years of Fair Representations: Challenges and Opportunitiescited by 6
  9. 2024Many Perception Tasks are Highly Redundant Functions of their Input Datacited by 4
  10. 2024Tangent Space Causal Inference: Leveraging Vector Fields for Causal Discovery in Dynamical Systemscited by 4
  11. 2025Connecting Jensen-Shannon and Kullback-Leibler Divergences: A New Bound for Representation Learningcited by 4
  12. 2025Neural Mutual Information Estimation with Vector Copulascited by 4
25 more
  1. 2025FMMI: Flow Matching Mutual Information Estimationcited by 4
  2. 2024SOmegaI: Score-based O-INFORMATION Estimationcited by 3
  3. 2025Meta-Statistical Learning: Supervised Learning of Statistical Estimatorscited by 3
  4. 2025Diffusion Secant Alignment for Score-Based Density Ratio Estimationcited by 3
  5. 2025MMG: Mutual Information Estimation via the MMSE Gap in Diffusioncited by 3
  6. 2026A Minimum Variance Path Principle for Accurate and Stable Score-Based Density Ratio Estimationcited by 3
  7. 2024Small Contributions, Small Networks: Efficient Neural Network Pruning Based on Relative Importancecited by 2
  8. 2025Accurate Estimation of Mutual Information in High Dimensional Datacited by 2
  9. 2023Measuring Information in Text Explanationscited by 1
  10. 2024Fault Detection and Monitoring using a Data-Driven Information-Based Strategy: Method, Theory, and Applicationcited by 1
  11. 2024Improving Numerical Stability of Normalized Mutual Information Estimator on High Dimensionscited by 1
  12. 2025A Neural Difference-of-Entropies Estimator for Mutual Informationcited by 1
  13. 2025RFMI: Estimating Mutual Information on Rectified Flow for Text-to-Image Alignmentcited by 1
  14. 2025Curse of Slicing: Why Sliced Mutual Information is a Deceptive Measure of Statistical Dependencecited by 1
  15. 2025Multimodal Datasets with Controllable Mutual Informationcited by 1
  16. 2025MIST: Mutual Information Estimation Via Supervised Trainingcited by 1
  17. 2026One-Step Score-Based Density Ratio Estimationcited by 1
  18. 2026InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimatecited by 1
  19. 2024Establishing Deep InfoMax as an effective self-supervised learning methodology in materials informaticscited by 0
  20. 2024Mutual Information Multinomial Estimationcited by 0
  21. 2025Towards a pretrained deep learning estimator of the Linfoot informational correlationcited by 0
  22. 2026Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimationcited by 0
  23. 2026GRIP: Grounded Reasoning via Information-Restricted Premisescited by 0
  24. 2026When Does Text Inform? Benchmarking Information-Theoretic Metrics for Multimodal Time-Series Forecastingcited by 0
  25. 2026ALICE: In-context, Zero-shot, Mutual Information Estimationcited by 0