mlsanityData to 7 Oct 2026

Understanding Contrastive Learning Requires Incorporating Inductive Biases

arXiv 2202.14037 ↗First posted 28 Feb 2022cs.LG cs.AIcited by 96

Not on the map. Why?

Abstract (arXiv)

Contrastive learning is a popular form of self-supervised learning that encourages augmentations (views) of the same input to have more similar representations compared to augmentations of different inputs. Recent attempts to theoretically explain the success of contrastive learning on downstream classification tasks prove guarantees depending on properties of em augmentations and the value of em contrastive loss of representations. We demonstrate that such analyses, that ignore em inductive biases of the function class and training algorithm, cannot adequately explain the success of contrastive learning, even em provably leading to vacuous guarantees in some settings. Extensive experiments on image and text domains highlight the ubiquity of this problem -- different function classes and algorithms behave very differently on downstream tasks, despite having the same augmentations and contrastive losses. Theoretical analysis is presented for the class of linear representations, where incorporating inductive biases of the function class allows contrastive learning to work with less stringent conditions compared to prior analyses.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Builds on 45 of its 54 references that are in the map. First posted on arXiv 28 Feb 2022.

Builds on

45 of 54 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  9. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  12. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
33 more
  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  3. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  4. 2019Contrastive Multiview Codingcited by 1,297
  5. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  6. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  7. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  8. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  9. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  10. 2019On Mutual Information Maximization for Representation Learningcited by 315
  11. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  12. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  13. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  14. 2017How to Escape Saddle Points Efficientlycited by 284
  15. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  16. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  17. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  18. 2018An efficient framework for learning sentence representationscited by 197
  19. 2020Representation Learning via Invariant Causal Mechanismscited by 197
  20. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  21. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  22. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  23. 2020Intriguing Properties of Contrastive Lossescited by 129
  24. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  25. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  26. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  27. 2020Understanding and Improving Information Transfer in Multi-Task Learningcited by 113
  28. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  29. 2020A Mathematical Exploration of Why Language Models Help Solve Downstream Taskscited by 71
  30. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54
  31. 2021Investigating the Role of Negatives in Contrastive Representation Learningcited by 54
  32. 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
  33. 2020For self-supervised learning, Rationality implies generalization, provablycited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 33; 2024: 21; 2025: 16; 2026: 10.

96 papers.

Sort:

By area:

  1. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 223
  2. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  3. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  4. 2022Orchestra: Unsupervised Federated Learning via Globally Consistent Clusteringcited by 45
  5. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  6. 2023The SSL Interplay: Augmentations, Inductive Bias, and Generalizationcited by 40
  7. 2022Do More Negative Samples Necessarily Hurt in Contrastive Learning?cited by 39
  8. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 39
  9. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  10. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 38
  11. 2024Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Modelscited by 38
  12. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
38 more
  1. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  2. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
  3. 2022Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functionscited by 32
  4. 2023Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppressioncited by 30
  5. 2023On the Generalization of Multi-modal Contrastive Learningcited by 30
  6. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
  7. 2023The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learningcited by 27
  8. 2023Survey of Social Bias in Vision-Language Modelscited by 26
  9. 2023Understanding Transferable Representation Learning and Zero-shot Transfer in CLIPcited by 26
  10. 2023Generalization Analysis for Contrastive Representation Learningcited by 25
  11. 2024Do Generated Data Always Help Contrastive Learning?cited by 25
  12. 2023Invariant Learning via Probability of Sufficient and Necessary Causescited by 22
  13. 2023On the Provable Advantage of Unsupervised Pretrainingcited by 21
  14. 2023Matrix Information Theory for Self-Supervised Learningcited by 21
  15. 2023Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regressioncited by 20
  16. 2024Investigating the Benefits of Projection Head for Representation Learningcited by 20
  17. 2024Position: Understanding LLMs Requires More Than Statistical Generalizationcited by 20
  18. 2022Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learningcited by 19
  19. 2022What shapes the loss landscape of self-supervised learning?cited by 19
  20. 2024On the Inductive Bias of Stacking Towards Improving Reasoningcited by 18
  21. 2023Information Flow in Self-Supervised Learningcited by 16
  22. 2024You Don't Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learningcited by 16
  23. 2023Optimal Sample Complexity of Contrastive Learningcited by 15
  24. 2023Identifiable Contrastive Learning with Automatic Feature Importance Discoverycited by 14
  25. 2022Integrating Prior Knowledge in Contrastive Learning with Kernelcited by 11
  26. 2022The Curious Case of Benign Memorizationcited by 11
  27. 2025Projection Head is Secretly an Information Bottleneckcited by 11
  28. 2024Efficient Stagewise Pretraining via Progressive Subnetworkscited by 10
  29. 2024A Unified Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capabilitycited by 9
  30. 2025On the Importance of Embedding Norms in Self-Supervised Learningcited by 9
  31. 2022Analyzing Data-Centric Properties for Graph Contrastive Learningcited by 7
  32. 2023Understanding Self-Supervised Pretraining with Part-Aware Representation Learningcited by 7
  33. 2023No Free Lunch in Self Supervised Representation Learningcited by 7
  34. 2024Causal Contrastive Learning for Counterfactual Regression Over Timecited by 7
  35. 2024The Hidden Pitfalls of the Cosine Similarity Losscited by 7
  36. 2024Collapse-Proof Non-Contrastive Self-Supervised Learningcited by 7
  37. 2024Understanding the Role of Equivariance in Self-supervised Learningcited by 7
  38. 2025Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Modelscited by 7

50 most cited of 96 All 96 (JSON)