mlsanityData to 7 Oct 2026

On Mutual Information Maximization for Representation Learning

arXiv 1907.13625 ↗First posted 31 Jul 2019cs.LG stat.MLcited by 315

Not on the map. Why?

Abstract (arXiv)

Many recent methods for unsupervised or self-supervised representation learning train feature extractors by maximizing an estimate of the mutual information (MI) between different views of the data. This comes with several immediate problems: For example, MI is notoriously hard to estimate, and using it as an objective for representation learning may lead to highly entangled representations due to its invariance under arbitrary invertible transformations. Nevertheless, these methods have been repeatedly shown to excel in practice. In this paper we argue, and provide empirical evidence, that the success of these methods cannot be attributed to the properties of MI alone, and that they strongly depend on the inductive bias in both the choice of feature extractor architectures and the parametrization of the employed MI estimators. Finally, we establish a connection to deep metric learning and argue that this interpretation may be a plausible explanation for the success of the recently introduced methods.

Key facts. Cited by 315 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 77; 2022: 56; 2023: 58; 2024: 41; 2025: 50; 2026: 33.

315 papers.

Sort:

By area:

  1. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  2. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  3. 2021Graph Self-Supervised Learning: A Surveycited by 289
  4. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  5. 2023A Cookbook of Self-Supervised Learningcited by 248
  6. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  7. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  8. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  9. 2021Adversarial Graph Augmentation to Improve Graph Contrastive Learningcited by 200
  10. 2021Self-Supervised Learning of Graph Neural Networks: A Unified Reviewcited by 184
  11. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  12. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
38 more
  1. 2021Self-supervised Learning on Graphs: Contrastive, Generative,or Predictivecited by 136
  2. 2021An Empirical Study of Graph Contrastive Learningcited by 113
  3. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  4. 2021Neural Transformation Learning for Deep Anomaly Detection Beyond Imagescited by 98
  5. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  6. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  7. 2022Concept Bottleneck Model with Additional Unsupervised Conceptscited by 79
  8. 2021Self-Supervised Learning with Kernel Dependence Maximizationcited by 77
  9. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
  10. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  11. 2021Multi-Scale Contrastive Siamese Networks for Self-Supervised Graph Representation Learningcited by 70
  12. 2022Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?cited by 67
  13. 2022Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embeddingcited by 67
  14. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  15. 2021TCL: Transformer-based Dynamic Graph Modelling via Contrastive Learningcited by 65
  16. 2021Category Contrast for Unsupervised Domain Adaptation in Visual Taskscited by 65
  17. 2021MoCL: Data-driven Molecular Fingerprint via Knowledge-aware Contrastive Learning from Molecular Graphcited by 65
  18. 2022Audio Self-supervised Learning: A Surveycited by 65
  19. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  20. 2023Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusioncited by 62
  21. 2022Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Frameworkcited by 59
  22. 2022Factorizing Knowledge in Neural Networkscited by 59
  23. 2022Visual Recognition with Deep Nearest Centroidscited by 59
  24. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  25. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 57
  26. 2021Investigating the Role of Negatives in Contrastive Representation Learningcited by 54
  27. 2021Shape or Texture: Understanding Discriminative Features in CNNscited by 52
  28. 2022Semi-Supervised and Unsupervised Deep Visual Learning: A Surveycited by 52
  29. 2021RGB-D Saliency Detection via Cascaded Mutual Information Minimizationcited by 50
  30. 2025Think Before Recommend: Unleashing the Latent Reasoning Power for Sequential Recommendationcited by 50
  31. 2021How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?cited by 49
  32. 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
  33. 2021A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectivescited by 41
  34. 2022Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learningcited by 40
  35. 2022Parametric Information Maximization for Generalized Category Discoverycited by 39
  36. 2021Dict-BERT: Enhancing Language Model Pre-training with Dictionarycited by 38
  37. 2022Self-Supervised Learning with an Information Maximization Criterioncited by 38
  38. 2021Prototypical Graph Contrastive Learningcited by 37

50 most cited of 315 All 315 (JSON)