Towards a Rigorous Analysis of Mutual Information in Contrastive Learning
Not on the map. Why?
Abstract (arXiv)
Contrastive learning has emerged as a cornerstone in recent achievements of unsupervised representation learning. Its primary paradigm involves an instance discrimination task with a mutual information loss. The loss is known as InfoNCE and it has yielded vital insights into contrastive learning through the lens of mutual information analysis. However, the estimation of mutual information can prove challenging, creating a gap between the elegance of its mathematical foundation and the complexity of its estimation. As a result, drawing rigorous insights or conclusions from mutual information analysis becomes intricate. In this study, we introduce three novel methods and a few related theorems, aimed at enhancing the rigor of mutual information analysis. Despite their simplicity, these methods can carry substantial utility. Leveraging these approaches, we reassess three instances of contrastive learning analysis, illustrating their capacity to facilitate deeper comprehension or to rectify pre-existing misconceptions. Specifically, we investigate small batch size, mutual information as a measure, and the InfoMin principle.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 61 of its 72 references that are in the map. First posted on arXiv 30 Aug 2023.
Builds on
61 of 72 references.
References · 61
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
- 2020Exploring Simple Siamese Representation Learningcited by 2,866
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
- 2020Supervised Contrastive Learningcited by 2,829
38 more
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
- 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
- 2016Density estimation using Real NVPcited by 2,042
- 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
- 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
- 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
- 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
- 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
- 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
- 2016Understanding intermediate layers using linear classifier probescited by 1,353
- 2019Contrastive Multiview Codingcited by 1,301
- 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
- 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
- 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
- 2017Deeper, Broader and Artier Domain Generalizationcited by 986
- 2020What Makes for Good Views for Contrastive Learning?cited by 846
- 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
- 2021Going deeper with Image Transformerscited by 764
- 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
- 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
- 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
- 2019On Variational Bounds of Mutual Informationcited by 559
- 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
- 2020Hard Negative Mixing for Contrastive Learningcited by 402
- 2020Understanding the Behaviour of Contrastive Losscited by 398
- 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
- 2019On Mutual Information Maximization for Representation Learningcited by 316
- 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
- 2020What Should Not Be Contrastive in Contrastive Learningcited by 226
- 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
- 2020Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biasescited by 143
- 2021Decoupled Contrastive Learningcited by 143
- 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
- 2020Intriguing Properties of Contrastive Lossescited by 130
- 2020What shapes feature representations? Exploring datasets, architectures, and trainingcited by 126
- 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2024: 2.
2 papers.