# On Mutual Information in Contrastive Learning for Visual Representations

arXiv: https://arxiv.org/abs/2005.13149 · HTML: https://mlsanity.com/paper/2005.13149/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2020.

## Built on by (most cited first, 30 of 54)

- [Exploring Cross-Image Pixel Contrast for Semantic Segmentation](https://mlsanity.com/paper/2101.11939/) (2021)
- [Factorized Contrastive Learning: Going Beyond Multi-view Redundancy](https://mlsanity.com/paper/2306.05268/) (2023)
- [Crafting Better Contrastive Views for Siamese Representation Learning](https://mlsanity.com/paper/2202.03278/) (2022)
- [Machine Unlearning for Image-to-Image Generative Models](https://mlsanity.com/paper/2402.00351/) (2024)
- [Contrastive Multimodal Fusion with TupleInfoNCE](https://mlsanity.com/paper/2107.02575/) (2021)
- [InfoNCE: Identifying the Gap Between Theory and Practice](https://mlsanity.com/paper/2407.00143/) (2024)
- [Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning](https://mlsanity.com/paper/2207.07635/) (2022)
- [Rethinking the Augmentation Module in Contrastive Learning: Learning Hierarchical Augmentation Invariance with Expanded Views](https://mlsanity.com/paper/2206.00227/) (2022)
- [Improving Self-Supervised Learning by Characterizing Idealized Representations](https://mlsanity.com/paper/2209.06235/) (2022)
- [Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learning](https://mlsanity.com/paper/2106.02866/) (2021)
- [Temperature as Uncertainty in Contrastive Learning](https://mlsanity.com/paper/2110.04403/) (2021)
- [Robust Representation Learning via Perceptual Similarity Metrics](https://mlsanity.com/paper/2106.06620/) (2021)
- [Finding Order in Chaos: A Novel Data Augmentation Method for Time Series in Contrastive Learning](https://mlsanity.com/paper/2309.13439/) (2023)
- [Composition-contrastive Learning for Sentence Embeddings](https://mlsanity.com/paper/2307.07380/) (2023)
- [Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective](https://mlsanity.com/paper/2203.03871/) (2022)
- [Learning Representations with Contrastive Self-Supervised Learning for Histopathology Applications](https://mlsanity.com/paper/2112.05760/) (2021)
- [Distance-based Hyperspherical Classification for Multi-source Open-Set Domain Adaptation](https://mlsanity.com/paper/2107.02067/) (2021)
- [Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling](https://mlsanity.com/paper/2110.07198/) (2021)
- [Consensus Learning for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2206.02583/) (2022)
- [Self-Supervised Learning of Representations for Space Generates Multi-Modular Grid Cells](https://mlsanity.com/paper/2311.02316/) (2023)
- [Invariant Anomaly Detection under Distribution Shifts: A Causal Perspective](https://mlsanity.com/paper/2312.14329/) (2023)
- [Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations](https://mlsanity.com/paper/2406.09366/) (2024)
- [Click-through Rate Prediction with Auto-Quantized Contrastive Learning](https://mlsanity.com/paper/2109.13921/) (2021)
- [CLIP-Lite: Information Efficient Visual Representation Learning with Language Supervision](https://mlsanity.com/paper/2112.07133/) (2021)
- [miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddings](https://mlsanity.com/paper/2211.04928/) (2022)
- [A Mutual Information Perspective on Federated Contrastive Learning](https://mlsanity.com/paper/2405.02081/) (2024)
- [Surrogate- and invariance-boosted contrastive learning for data-scarce applications in science](https://mlsanity.com/paper/2110.08406/) (2021)
- [Evaluating Self-Supervised Learning via Risk Decomposition](https://mlsanity.com/paper/2302.03068/) (2023)
- [VOLTA: an Environment-Aware Contrastive Cell Representation Learning for Histopathology](https://mlsanity.com/paper/2303.04696/) (2023)
- [Variational Inference Failures Under Model Symmetries: Permutation Invariant Posteriors for Bayesian Neural Networks](https://mlsanity.com/paper/2408.05496/) (2024)

## Abstract (arXiv)

In recent years, several unsupervised, "contrastive" learning algorithms in vision have been shown to learn representations that perform remarkably well on transfer tasks. We show that this family of algorithms maximizes a lower bound on the mutual information between two or more "views" of an image where typical views come from a composition of image augmentations. Our bound generalizes the InfoNCE objective to support negative sampling from a restricted region of "difficult" contrasts. We find that the choice of negative samples and views are critical to the success of these algorithms. Reformulating previous learning objectives in terms of mutual information also simplifies and stabilizes them. In practice, our new objectives yield representations that outperform those learned with previous approaches for transfer to classification, bounding box detection, instance segmentation, and keypoint detection. % experiments show that choosing more difficult negative samples results in a stronger representation, outperforming those learned with IR, LA, and CMC in classification, bounding box detection, instance segmentation, and keypoint detection. The mutual information framework provides a unifying comparison of approaches to contrastive learning and uncovers the choices that impact representation learning.