mlsanityData to 7 Oct 2026

CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information

arXiv 2006.12013 ↗First posted 22 Jun 2020cs.LG stat.MLcited by 233

Not on the map. Why?

Abstract (arXiv)

Mutual information (MI) minimization has gained considerable interests in various machine learning tasks. However, estimating and minimizing MI in high-dimensional spaces remains a challenging problem, especially when only samples, rather than distribution forms, are accessible. Previous works mainly focus on MI lower bound approximation, which is not applicable to MI minimization problems. In this paper, we propose a novel Contrastive Log-ratio Upper Bound (CLUB) of mutual information. We provide a theoretical analysis of the properties of CLUB and its variational approximation. Based on this upper bound, we introduce a MI minimization training scheme and further accelerate it with a negative sampling strategy. Simulation studies on Gaussian distributions show the reliable estimation ability of CLUB. Real-world MI minimization experiments, including domain adaptation and information bottleneck, demonstrate the effectiveness of the proposed method. The code is at https://github.com/Linear95/CLUB.

Key facts. Cited by 233 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 34; 2023: 40; 2024: 52; 2025: 48; 2026: 38.

233 papers.

Sort:

By area:

  1. 2021Learning to Diversify for Single Domain Generalizationcited by 170
  2. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 130
  3. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  4. 2023Collaborative Perception in Autonomous Driving: Methods, Datasets and Challengescited by 87
  5. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  6. 2021FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoderscited by 75
  7. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
  8. 2024AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encodingcited by 67
  9. 2021VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversioncited by 63
  10. 2022Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Featurescited by 63
  11. 2023Time Series Contrastive Learning with Information-Aware Augmentationscited by 52
  12. 2023Domain Watermark: Effective and Harmless Dataset Copyright Protection is Closed at Handcited by 52
38 more
  1. 2024Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Predictioncited by 41
  2. 2021A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representationscited by 37
  3. 2023Multimodal Variational Auto-encoder based Audio-Visual Segmentationcited by 37
  4. 2021Selective Regression Under Fairness Criteriacited by 34
  5. 2023DisDiff: Unsupervised Disentanglement of Diffusion Probabilistic Modelscited by 30
  6. 2021Gradient-based Bayesian Experimental Design for Implicit Models using Mutual Information Lower Boundscited by 26
  7. 2023RPEFlow: Multimodal Fusion of RGB-PointCloud-Event for Joint Optical Flow and Scene Flow Estimationcited by 26
  8. 2022Temporal Feature Alignment and Mutual Information Maximization for Video-Based Human Pose Estimationcited by 24
  9. 2021Privacy-Preserving Representation Learning on Graphs: A Mutual Information Perspectivecited by 23
  10. 2022PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaborationcited by 23
  11. 2021Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCEcited by 22
  12. 2024Visual Neural Decoding via Improved Visual-EEG Semantic Consistencycited by 21
  13. 2022A Differential Entropy Estimator for Training Neural Networkscited by 20
  14. 2023DiME: Maximizing Mutual Information by a Difference of Matrix-Based Entropiescited by 20
  15. 2021DisenKGAT: Knowledge Graph Embedding with Disentangled Graph Attention Networkcited by 19
  16. 2021Layer-wise Model Pruning based on Mutual Informationcited by 19
  17. 2023Theory of Mind as Intrinsic Motivation for Multi-Agent Reinforcement Learningcited by 19
  18. 2021Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimizationcited by 18
  19. 2021Auto IV: Counterfactual Prediction via Automatic Instrumental Variable Decompositioncited by 18
  20. 2022Learning Disentangled Representations for Counterfactual Regression via Mutual Information Minimizationcited by 18
  21. 2022Decoupling Local and Global Representations of Time Seriescited by 17
  22. 2023Context Shift Reduction for Offline Meta-Reinforcement Learningcited by 17
  23. 2024Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learningcited by 16
  24. 2024EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusioncited by 16
  25. 2024Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworkscited by 15
  26. 2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approachcited by 15
  27. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
  28. 2022Learning Disentangled Textual Representations via Statistical Measures of Similaritycited by 14
  29. 2022Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversioncited by 14
  30. 2022Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributionscited by 14
  31. 2022Learning Disentangled Representations of Negation and Uncertaintycited by 13
  32. 2023Towards Open Temporal Graph Neural Networkscited by 13
  33. 2024Inf2Guard: An Information-Theoretic Framework for Learning Privacy-Preserving Representations against Inference Attackscited by 13
  34. 2024Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learningcited by 13
  35. 2022Measures of Information Reflect Memorization Patternscited by 12
  36. 2023Toward Fairness in Text Generation via Mutual Information Minimization based on Importance Samplingcited by 12
  37. 2025STiL: Semi-supervised Tabular-Image Learning for Comprehensive Task-Relevant Information Exploration in Multimodal Classificationcited by 12
  38. 2022Controllable Augmentations for Video Representation Learningcited by 10

50 most cited of 233 All 233 (JSON)