mlsanityData to 7 Oct 2026

Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization

arXiv 2107.01131 ↗First posted 2 Jul 2021stat.ML cs.AI cs.ITcited by 18

Not on the map. Why?

Abstract (arXiv)

Successful applications of InfoNCE and its variants have popularized the use of contrastive variational mutual information (MI) estimators in machine learning. While featuring superior stability, these estimators crucially depend on costly large-batch training, and they sacrifice bound tightness for variance reduction. To overcome these limitations, we revisit the mathematics of popular variational MI bounds from the lens of unnormalized statistical modeling and convex optimization. Our investigation not only yields a new unified theoretical framework encompassing popular variational MI bounds but also leads to a novel, simple, and powerful contrastive MI estimator named as FLO. Theoretically, we show that the FLO estimator is tight, and it provably converges under stochastic gradient descent. Empirically, our FLO estimator overcomes the limitations of its predecessors and learns more efficiently. The utility of FLO is verified using an extensive set of benchmarks, which also reveals the trade-offs in practical MI estimation.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 43 of its 116 references that are in the map. First posted on arXiv 2 Jul 2021.

Builds on

43 of 116 references.

References · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2013Auto-Encoding Variational Bayescited by 3,980
  8. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  9. 2016node2vec: Scalable Feature Learning for Networkscited by 1,913
  10. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  11. 2019Contrastive Multiview Codingcited by 1,297
  12. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
31 more
  1. 2016Deep Variational Information Bottleneckcited by 915
  2. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  3. 2018MINE: Mutual Information Neural Estimationcited by 680
  4. 2018Mitigating Unwanted Biases with Adversarial Learningcited by 663
  5. 2019On Variational Bounds of Mutual Informationcited by 550
  6. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  7. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  8. 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 460
  9. 2018AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Biascited by 319
  10. 2019On Mutual Information Maximization for Representation Learningcited by 315
  11. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  12. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  13. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 233
  14. 2014Predictive Entropy Search for Efficient Global Optimization of Black-box Functionscited by 201
  15. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  16. 2020A Theory of Usable Information Under Computational Constraintscited by 153
  17. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  18. 2019Variational Bayesian Optimal Experimental Designcited by 116
  19. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
  20. 2021Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Designcited by 103
  21. 2018Learning Controllable Fair Representationscited by 83
  22. 2017On Nesting Monte Carlo Estimatorscited by 82
  23. 2019A Unified Stochastic Gradient Approach to Designing Bayesian-Optimal Experimentscited by 58
  24. 2020Bayesian Experimental Design for Implicit Models by Mutual Information Neural Estimationcited by 56
  25. 2014Efficient Estimation of Mutual Information for Strongly Dependent Variablescited by 46
  26. 2021Controllable Guarantees for Fair Outcomes via Contrastive Information Estimationcited by 45
  27. 2021Gradient-based Bayesian Experimental Design for Implicit Models using Mutual Information Lower Boundscited by 26
  28. 2020Sequential Bayesian Experimental Design for Implicit Models via Mutual Informationcited by 23
  29. 2013Predictive information in a sensory populationcited by 22
  30. 2016Demystifying Fixed k-Nearest Neighbor Information Estimatorscited by 14
  31. 2020Mutual Information Gradient Estimation for Representation Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 1; 2023: 6; 2024: 3; 2025: 4; 2026: 2.

18 papers.

Sort:

By area:

  1. 2023A Cookbook of Self-Supervised Learningcited by 248
  2. 2023Modern Bayesian Experimental Designcited by 121
  3. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
  4. 2021On the Surrogate Gap between Contrastive and Supervised Lossescited by 25
  5. 2023Understanding Contrastive Learning via Distributionally Robust Optimizationcited by 21
  6. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  7. 2023Max-Sliced Mutual Informationcited by 10
  8. 2022RenyiCL: Contrastive Representation Learning with Skew Renyi Divergencecited by 8
  9. 2023On the Effectiveness of Hybrid Mutual Information Estimationcited by 6
  10. 2025The Hidden Link Between RLHF and Contrastive Learningcited by 5
  11. 2025Neural Mutual Information Estimation with Vector Copulascited by 4
  12. 2021Quantifying the Task-Specific Information in Text-Based Classificationscited by 3
6 more
  1. 2025Accurate Estimation of Mutual Information in High Dimensional Datacited by 2
  2. 2024A Comprehensive Case Study on the Performance of Machine Learning Methods on the Classification of Solar Panel Electroluminescence Imagescited by 1
  3. 2024Skill-aware Mutual Information Optimisation for Generalisation in Reinforcement Learningcited by 0
  4. 2025A Deep Bayesian Nonparametric Framework for Robust Mutual Information Estimationcited by 0
  5. 2026Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learningcited by 0
  6. 2026CLaST: Context-aware Contrastive VAE for Probabilistic Time Series Forecastingcited by 0