# Contrastive learning, multi-view redundancy, and linear models

arXiv: https://arxiv.org/abs/2008.10150 · HTML: https://mlsanity.com/paper/2008.10150/

## Key facts

- Cited by 146 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Aug 2020.

## Built on by (most cited first, 30 of 146)

- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style](https://mlsanity.com/paper/2106.04619/) (2021)
- [Understanding self-supervised Learning Dynamics without Contrastive Pairs](https://mlsanity.com/paper/2102.06810/) (2021)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods](https://mlsanity.com/paper/2205.11508/) (2022)
- [Self-supervised Learning is More Robust to Dataset Imbalance](https://mlsanity.com/paper/2110.05025/) (2021)
- [A Kernel-Based View of Language Model Fine-Tuning](https://mlsanity.com/paper/2210.05643/) (2022)
- [Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learning](https://mlsanity.com/paper/2105.15134/) (2021)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap](https://mlsanity.com/paper/2203.13457/) (2022)
- [Understanding Contrastive Learning Requires Incorporating Inductive Biases](https://mlsanity.com/paper/2202.14037/) (2022)
- [Divide and Contrast: Self-supervised Learning from Uncurated Data](https://mlsanity.com/paper/2105.08054/) (2021)
- [WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training](https://mlsanity.com/paper/2103.06561/) (2021)
- [Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework](https://mlsanity.com/paper/2302.12247/) (2023)
- [Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning](https://mlsanity.com/paper/2106.09226/) (2021)
- [Factorized Contrastive Learning: Going Beyond Multi-view Redundancy](https://mlsanity.com/paper/2306.05268/) (2023)
- [Multi-View Causal Representation Learning with Partial Observability](https://mlsanity.com/paper/2311.04056/) (2023)
- [Generative and Contrastive Self-Supervised Learning for Graph Anomaly Detection](https://mlsanity.com/paper/2108.09896/) (2021)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [Investigating the Role of Negatives in Contrastive Representation Learning](https://mlsanity.com/paper/2106.09943/) (2021)
- [The Power of Contrast for Feature Learning: A Theoretical Analysis](https://mlsanity.com/paper/2110.02473/) (2021)
- [Lossy Compression for Lossless Prediction](https://mlsanity.com/paper/2106.10800/) (2021)
- [Making Linear MDPs Practical via Contrastive Representation Learning](https://mlsanity.com/paper/2207.07150/) (2022)
- [Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning](https://mlsanity.com/paper/2102.06866/) (2021)
- [Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulations](https://mlsanity.com/paper/2204.02683/) (2022)
- [A Theoretical Study of Inductive Biases in Contrastive Learning](https://mlsanity.com/paper/2211.14699/) (2022)
- [The SSL Interplay: Augmentations, Inductive Bias, and Generalization](https://mlsanity.com/paper/2302.02774/) (2023)
- [The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning](https://mlsanity.com/paper/2205.06226/) (2022)
- [Understanding Latent Correlation-Based Multiview Learning and Self-Supervision: An Identifiability Perspective](https://mlsanity.com/paper/2106.07115/) (2021)

## Abstract (arXiv)

Self-supervised learning is an empirically successful approach to unsupervised learning based on creating artificial supervised learning problems. A popular self-supervised approach to representation learning is contrastive learning, which leverages naturally occurring pairs of similar and dissimilar data points, or multiple views of the same data. This work provides a theoretical analysis of contrastive learning in the multi-view setting, where two views of each datum are available. The main result is that linear functions of the learned representations are nearly optimal on downstream prediction tasks whenever the two views provide redundant information about the label.