mlsanityData to 14 Oct 2026

Understanding the Limitations of Variational Mutual Information Estimators

arXiv 1910.06222 ↗First posted 14 Oct 2019cs.LG cs.IT math.ITcited by 140

Not on the map. Why?

Abstract (arXiv)

Variational approaches based on neural networks are showing promise for estimating mutual information (MI) between high dimensional variables. However, they can be difficult to use in practice due to poorly understood bias/variance tradeoffs. We theoretically show that, under some conditions, estimators such as MINE exhibit variance that could grow exponentially with the true amount of underlying MI. We also empirically demonstrate that existing estimators fail to satisfy basic self-consistency properties of MI, such as data processing and additivity under independence. Based on a unified perspective of variational approaches, we develop a new estimator that focuses on variance reduction. Empirical results on standard benchmark tasks demonstrate that our proposed estimator exhibits improved bias-variance trade-offs on standard benchmark tasks.

Key facts. Cited by 140 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 32; 2022: 23; 2023: 21; 2024: 27; 2025: 26; 2026: 11.

140 papers.

Sort:

By area:

  1. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  2. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  3. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 75
  4. 2022Rethinking Minimal Sufficient Representation in Contrastive Learningcited by 69
  5. 2021Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoodscited by 61
  6. 2021Density Ratio Estimation via Infinitesimal Classificationcited by 61
  7. 2021DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimationcited by 48
  8. 2021Negative Data Augmentationcited by 46
  9. 2022Discrete Contrastive Diffusion for Cross-Modal Music and Image Generationcited by 44
  10. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 37
  11. 2023Beyond Normal: On the Evaluation of Mutual Information Estimatorscited by 37
  12. 2021Featurized Density Ratio Estimationcited by 33
38 more
  1. 2023MINDE: Mutual Information Neural Diffusion Estimationcited by 31
  2. 2021Which Mutual-Information Representation Learning Objectives are Sufficient for Control?cited by 29
  3. 2021Self-supervised Representation Learning with Relative Predictive Codingcited by 28
  4. 2021Multimodal Representation Learning via Maximization of Local Mutual Informationcited by 27
  5. 2021Gradient-based Bayesian Experimental Design for Implicit Models using Mutual Information Lower Boundscited by 27
  6. 2021Decomposed Mutual Information Estimation for Contrastive Representation Learningcited by 25
  7. 2021On the Surrogate Gap between Contrastive and Supervised Lossescited by 25
  8. 2021Clustering by Maximizing Mutual Information Across Viewscited by 23
  9. 2024HiLo: A Learning Framework for Generalized Category Discovery Robust to Domain Shiftscited by 23
  10. 2021Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCEcited by 22
  11. 2024Mutual Information Estimation via Normalizing Flowscited by 22
  12. 2023Estimating Conditional Mutual Information for Dynamic Feature Selectioncited by 21
  13. 2023Understanding Contrastive Learning via Distributionally Robust Optimizationcited by 21
  14. 2024Approximating mutual information of high-dimensional variables using learned representationscited by 21
  15. 2022A Differential Entropy Estimator for Training Neural Networkscited by 20
  16. 2022Self-Supervision Can Be a Good Few-Shot Learnercited by 19
  17. 2021Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimizationcited by 18
  18. 2023Mutual Information Estimation via f-Divergence and Data Derangementscited by 17
  19. 2022Policy-Based Bayesian Experimental Design for Non-Differentiable Implicit Modelscited by 16
  20. 2021Improving Multimodal fusion via Mutual Dependency Maximisationcited by 15
  21. 2021Boosting Semi-supervised Image Segmentation with Global and Local Mutual Information Regularizationcited by 14
  22. 2022Learning Disentangled Textual Representations via Statistical Measures of Similaritycited by 14
  23. 2022COIN: Co-Cluster Infomax for Bipartite Graphscited by 14
  24. 2022Copula Density Neural Estimationcited by 13
  25. 2025Density Ratio Estimation with Conditional Probability Pathscited by 13
  26. 2023STERLING: Synergistic Representation Learning on Bipartite Graphscited by 12
  27. 2021Information-Theoretic Analysis of Epistemic Uncertainty in Bayesian Meta-learningcited by 10
  28. 2021AID-Purifier: A Light Auxiliary Network for Boosting Adversarial Defensecited by 10
  29. 2023Max-Sliced Mutual Informationcited by 10
  30. 2024TexShape: Information Theoretic Sentence Embedding for Language Modelscited by 10
  31. 2024A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasetscited by 10
  32. 2022RenyiCL: Contrastive Representation Learning with Skew Renyi Divergencecited by 8
  33. 2023Scalable Infomin Learningcited by 8
  34. 2023Mini-Batch Optimization of Contrastive Losscited by 8
  35. 2024f-Divergence Based Classification: Beyond the Use of Cross-Entropycited by 8
  36. 2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 8
  37. 2024Information Theoretic Text-to-Image Alignmentcited by 8
  38. 2024Automated Discovery of Pairwise Interactions from Unstructured Datacited by 8

50 most cited of 140 All 140 (JSON)