mlsanityData to 14 Oct 2026

Formal Limitations on the Measurement of Mutual Information

arXiv 1811.04251 ↗First posted 10 Nov 2018cs.IT cs.LG math.ITcited by 189

Not on the map. Why?

Abstract (arXiv)

Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confidence lower bound on mutual information estimated from N samples cannot be larger than O(ln N ).

Key facts. Cited by 189 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 42; 2022: 31; 2023: 30; 2024: 30; 2025: 35; 2026: 20.

189 papers.

Sort:

By area:

  1. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
  2. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 190
  3. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 89
  4. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 81
  5. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  6. 2022Rethinking Minimal Sufficient Representation in Contrastive Learningcited by 69
  7. 2021Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoodscited by 61
  8. 2021Density Ratio Estimation via Infinitesimal Classificationcited by 61
  9. 2023Equivariant Flow Matching with Hybrid Probability Transportcited by 60
  10. 2021Shape or Texture: Understanding Discriminative Features in CNNscited by 52
  11. 2021Lossy Compression for Lossless Predictioncited by 52
  12. 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
38 more
  1. 2021Generalization Bounds For Meta-Learning: An Information-Theoretic Analysiscited by 45
  2. 2022Robust Contrastive Learning against Noisy Viewscited by 45
  3. 2022Self-Supervised Learning with an Information Maximization Criterioncited by 38
  4. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 37
  5. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  6. 2023Beyond Normal: On the Evaluation of Mutual Information Estimatorscited by 37
  7. 2022Dataset Inference for Self-Supervised Modelscited by 36
  8. 2024TimeX++: Learning Time-Series Explanations with Information Bottleneckcited by 33
  9. 2023MINDE: Mutual Information Neural Diffusion Estimationcited by 31
  10. 2021CLCC: Contrastive Learning for Color Constancycited by 29
  11. 2021Which Mutual-Information Representation Learning Objectives are Sufficient for Control?cited by 29
  12. 2021Self-supervised Representation Learning with Relative Predictive Codingcited by 28
  13. 2021Decomposed Mutual Information Estimation for Contrastive Representation Learningcited by 25
  14. 2021On the Surrogate Gap between Contrastive and Supervised Lossescited by 25
  15. 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
  16. 2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignmentcited by 25
  17. 2021Clustering by Maximizing Mutual Information Across Viewscited by 23
  18. 2021Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCEcited by 22
  19. 2021A Bayesian Framework for Information-Theoretic Probingcited by 22
  20. 2024Mutual Information Estimation via Normalizing Flowscited by 22
  21. 2023DiME: Maximizing Mutual Information by a Difference of Matrix-Based Entropiescited by 21
  22. 2023Multimodal Molecular Pretraining via Modality Blendingcited by 21
  23. 2024Approximating mutual information of high-dimensional variables using learned representationscited by 21
  24. 2022A Differential Entropy Estimator for Training Neural Networkscited by 20
  25. 2021X-GOAL: Multiplex Heterogeneous Graph Prototypical Contrastive Learningcited by 19
  26. 2023Information Bottleneck Analysis of Deep Neural Networks via Lossy Compressioncited by 19
  27. 2021Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' Outputscited by 18
  28. 2021Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimizationcited by 18
  29. 2023Mutual Information Estimation via f-Divergence and Data Derangementscited by 17
  30. 2025Toward Efficient Exploration by Large Language Model Agentscited by 17
  31. 2021Improving Multimodal fusion via Mutual Dependency Maximisationcited by 15
  32. 2021Boosting Semi-supervised Image Segmentation with Global and Local Mutual Information Regularizationcited by 14
  33. 2022Learning Disentangled Textual Representations via Statistical Measures of Similaritycited by 14
  34. 2021Tensor networks and efficient descriptions of classical datacited by 13
  35. 2023The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learningcited by 13
  36. 2024f-MICL: Understanding and Generalizing InfoNCE-based Contrastive Learningcited by 13
  37. 2021M2IOSR: Maximal Mutual Information Open Set Recognitioncited by 12
  38. 2023Quantifying the redundancy between prosody and textcited by 12

50 most cited of 189 All 189 (JSON)