mlsanityData to 7 Oct 2026

Efficient Estimation of Mutual Information for Strongly Dependent Variables

arXiv 1411.2003 ↗First posted 7 Nov 2014cs.IT math.IT physics.data-ancited by 46

Not on the map. Why?

Abstract (arXiv)

We demonstrate that a popular class of nonparametric mutual information (MI) estimators based on k-nearest-neighbor graphs requires number of samples that scales exponentially with the true MI. Consequently, accurate estimation of MI between two strongly dependent variables is possible only for prohibitively large sample size. This important yet overlooked shortcoming of the existing estimators is due to their implicit reliance on local uniformity of the underlying joint distribution. We introduce a new estimator that is robust to local non-uniformity, works well with limited data, and is able to capture relationship strengths over many orders of magnitude. We demonstrate the superior performance of the proposed estimator on both synthetic and real-world data.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Nov 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 4; 2023: 11; 2024: 8; 2025: 9; 2026: 5.

46 papers.

Sort:

By area:

  1. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  2. 2021Causal Influence Detection for Improving Efficiency in Reinforcement Learningcited by 70
  3. 2021Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identificationcited by 50
  4. 2023Beyond Normal: On the Evaluation of Mutual Information Estimatorscited by 37
  5. 2023MINDE: Mutual Information Neural Diffusion Estimationcited by 31
  6. 2021On the Surrogate Gap between Contrastive and Supervised Lossescited by 25
  7. 2021Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCEcited by 22
  8. 2024Approximating mutual information of high-dimensional variables using learned representationscited by 21
  9. 2021Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimizationcited by 18
  10. 2023VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distributioncited by 17
  11. 2023Sample Complexity of Probability Divergences under Group Symmetrycited by 13
  12. 2023Assessing Neural Network Representations During Training Using Noise-Resilient Diffusion Spectral Entropycited by 13
34 more
  1. 2022Network Binarization via Contrastive Learningcited by 12
  2. 2023Robust Concept Erasure via Kernelized Rate-Distortion Maximizationcited by 10
  3. 2024A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasetscited by 10
  4. 2021Bird's Eye: Probing for Linguistic Graph Structures with a Simple Information-Theoretic Approachcited by 9
  5. 2025L2M: Mutual Information Scaling Law for Long-Context Language Modelingcited by 8
  6. 2022miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddingscited by 7
  7. 2021MIND: Inductive Mutual Information Estimation, A Convex Maximum-Entropy Copula Approachcited by 6
  8. 2023On the Effectiveness of Hybrid Mutual Information Estimationcited by 6
  9. 2021A Reverse Jensen Inequality Result with Application to Mutual Information Estimationcited by 5
  10. 2022A robust estimator of mutual information for deep learning interpretabilitycited by 4
  11. 2024Many Perception Tasks are Highly Redundant Functions of their Input Datacited by 4
  12. 2023A Distance Correlation-Based Approach to Characterize the Effectiveness of Recurrent Neural Networks for Time Series Forecastingcited by 3
  13. 2025MMG: Mutual Information Estimation via the MMSE Gap in Diffusioncited by 3
  14. 2022Trying to Outrun Causality with Machine Learning: Limitations of Model Explainability Techniques for Identifying Predictive Variablescited by 2
  15. 2023Towards a Rigorous Analysis of Mutual Information in Contrastive Learningcited by 2
  16. 2024An Information Theoretic Approach to Interaction-Grounded Learningcited by 2
  17. 2025INFO-SEDD: Continuous Time Markov Chains as Scalable Information Metrics Estimatorscited by 2
  18. 2025Accurate Estimation of Mutual Information in High Dimensional Datacited by 2
  19. 2025TELL-TALE: Task Efficient LLMs with Task Aware Layer Eliminationcited by 2
  20. 2023Safety Performance of Neural Networks in the Presence of Covariate Shiftcited by 1
  21. 2025Multimodal Datasets with Controllable Mutual Informationcited by 1
  22. 2026Forecastability as an Information-Theoretic Limit on Predictioncited by 1
  23. 2026Efficient Benchmarking Is Just Feature Selection and Multiple Regressioncited by 1
  24. 2021A Simple Generative Networkcited by 0
  25. 2024Multimodal Interpretable Data-Driven Models for Early Prediction of Antimicrobial Multidrug Resistance Using Multivariate Time-Seriescited by 0
  26. 2024Optimizing Privacy and Utility Tradeoffs for Group Interests Through Harmonizationcited by 0
  27. 2024Mutual Information Multinomial Estimationcited by 0
  28. 2024fastHDMI: Fast Mutual Information Estimation for High-Dimensional Datacited by 0
  29. 2025Causal-Aware Intelligent QoE Optimization for VR Interaction with Adaptive Keyframe Extractioncited by 0
  30. 2025ML-PWS: Estimating the Mutual Information Between Experimental Time Series Using Neural Networkscited by 0
  31. 2025Nonparametric Estimation of Joint Entropy via Partitioned Sample-Spacingcited by 0
  32. 2026TAPIOCA: Why Task- Aware Pruning Improves OOD model Capabilitycited by 0
  33. 2026NMINE: Normalized Mutual Information Neural Estimationcited by 0
  34. 2026ALICE: In-context, Zero-shot, Mutual Information Estimationcited by 0