mlsanityData to 14 Oct 2026

On Variational Bounds of Mutual Information

arXiv 1905.06922 ↗First posted 16 May 2019cs.LG stat.MLcited by 559

Not on the map. Why?

Abstract (arXiv)

Estimating and optimizing Mutual Information (MI) is core to many problems in machine learning; however, bounding MI in high dimensions is challenging. To establish tractable and scalable objectives, recent work has turned to variational bounds parameterized by neural networks, but the relationships and tradeoffs between these bounds remains unclear. In this work, we unify these recent developments in a single framework. We find that the existing variational lower bounds degrade when the MI is large, exhibiting either high bias or high variance. To address this problem, we introduce a continuum of lower bounds that encompasses previous bounds and flexibly trades off bias and variance. On high-dimensional, controlled problems, we empirically characterize the bias and variance of the bounds and their gradients and demonstrate the effectiveness of our new bounds for estimation and representation learning.

Key facts. Cited by 559 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 98; 2022: 98; 2023: 97; 2024: 98; 2025: 85; 2026: 83.

559 papers.

Sort:

By area:

  1. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  2. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
  3. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
  4. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  5. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  6. 2021Adversarial Graph Augmentation to Improve Graph Contrastive Learningcited by 200
  7. 2022Interpretable and Generalizable Graph Learning via Stochastic Attention Mechanismcited by 171
  8. 2021VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examplescited by 154
  9. 2021Deep Stable Learning for Out-Of-Distribution Generalizationcited by 153
  10. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 138
  11. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  12. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 131
38 more
  1. 2023Modern Bayesian Experimental Designcited by 125
  2. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  3. 2021An Empirical Study of Graph Contrastive Learningcited by 113
  4. 2021Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Designcited by 106
  5. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  6. 2023SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentationcited by 104
  7. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 89
  8. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  9. 2022What's Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoderscited by 82
  10. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 79
  11. 2021Self-Supervised Learning with Kernel Dependence Maximizationcited by 77
  12. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 75
  13. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  14. 2021Causal Influence Detection for Improving Efficiency in Reinforcement Learningcited by 71
  15. 2022Rethinking Minimal Sufficient Representation in Contrastive Learningcited by 69
  16. 2021TCL: Transformer-based Dynamic Graph Modelling via Contrastive Learningcited by 68
  17. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 68
  18. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 68
  19. 2021MoCL: Data-driven Molecular Fingerprint via Knowledge-aware Contrastive Learning from Molecular Graphcited by 65
  20. 2022Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Featurescited by 63
  21. 2021Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoodscited by 61
  22. 2021Density Ratio Estimation via Infinitesimal Classificationcited by 61
  23. 2023Identifiability Results for Multimodal Contrastive Learningcited by 61
  24. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 56
  25. 2021Temporal Predictive Coding For Model-Based Planning In Latent Spacecited by 55
  26. 2021Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergencecited by 54
  27. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 54
  28. 2021Shape or Texture: Understanding Discriminative Features in CNNscited by 52
  29. 2021Lossy Compression for Lossless Predictioncited by 52
  30. 2023Time Series Contrastive Learning with Information-Aware Augmentationscited by 52
  31. 2021Optimal Representations for Covariate Shiftcited by 51
  32. 2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 51
  33. 2021Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identificationcited by 50
  34. 2021How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?cited by 49
  35. 2022Interventions, Where and How? Experimental Design for Causal Models at Scalecited by 49
  36. 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
  37. 2024Machine Unlearning for Image-to-Image Generative Modelscited by 47
  38. 2024Exposing the Deception: Uncovering More Forgery Clues for Deepfake Detectioncited by 47

50 most cited of 559 All 559 (JSON)