mlsanityData to 7 Oct 2026

Contrastive learning, multi-view redundancy, and linear models

arXiv 2008.10150 ↗First posted 24 Aug 2020cs.LG stat.MLcited by 146

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning is an empirically successful approach to unsupervised learning based on creating artificial supervised learning problems. A popular self-supervised approach to representation learning is contrastive learning, which leverages naturally occurring pairs of similar and dissimilar data points, or multiple views of the same data. This work provides a theoretical analysis of contrastive learning in the multi-view setting, where two views of each datum are available. The main result is that linear functions of the learned representations are nearly optimal on downstream prediction tasks whenever the two views provide redundant information about the label.

Key facts. Cited by 146 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 36; 2023: 35; 2024: 17; 2025: 20; 2026: 10.

146 papers.

Sort:

By area:

  1. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  2. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  3. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  4. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  5. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  6. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  7. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 122
  8. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  9. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  10. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  11. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  12. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
38 more
  1. 2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
  2. 2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
  3. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  4. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 81
  5. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
  6. 2023Multi-View Causal Representation Learning with Partial Observabilitycited by 72
  7. 2021Generative and Contrastive Self-Supervised Learning for Graph Anomaly Detectioncited by 67
  8. 2022Audio Self-supervised Learning: A Surveycited by 65
  9. 2021Investigating the Role of Negatives in Contrastive Representation Learningcited by 54
  10. 2021The Power of Contrast for Feature Learning: A Theoretical Analysiscited by 53
  11. 2021Lossy Compression for Lossless Predictioncited by 51
  12. 2022Making Linear MDPs Practical via Contrastive Representation Learningcited by 49
  13. 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
  14. 2022Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulationscited by 44
  15. 2023The SSL Interplay: Augmentations, Inductive Bias, and Generalizationcited by 40
  16. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  17. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 38
  18. 2021Understanding Latent Correlation-Based Multiview Learning and Self-Supervision: An Identifiability Perspectivecited by 37
  19. 2022On the Power of Foundation Modelscited by 37
  20. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  21. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  22. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
  23. 2022Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functionscited by 32
  24. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 32
  25. 2023Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppressioncited by 30
  26. 2023Revisiting Discriminative vs. Generative Classifiers: Theory and Implicationscited by 29
  27. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
  28. 2025Understanding the Emergence of Multimodal Representation Alignmentcited by 28
  29. 2021Self-supervised Representation Learning with Relative Predictive Codingcited by 27
  30. 2023The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learningcited by 27
  31. 2021Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learningcited by 25
  32. 2021On the Surrogate Gap between Contrastive and Supervised Lossescited by 25
  33. 2023Generalization Analysis for Contrastive Representation Learningcited by 25
  34. 2021Towards Demystifying Representation Learning with Non-contrastive Self-supervisioncited by 23
  35. 2021Object-Aware Cropping for Self-Supervised Learningcited by 22
  36. 2022Augment with Care: Contrastive Learning for Combinatorial Problemscited by 22
  37. 2023A Graph-Theoretic Framework for Understanding Open-World Semi-Supervised Learningcited by 22
  38. 2023On the Provable Advantage of Unsupervised Pretrainingcited by 21

50 most cited of 146 All 146 (JSON)