mlsanityData to 14 Oct 2026

Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppression

arXiv 2305.16536 ↗First posted 25 May 2023cs.LG stat.MLcited by 30

Not on the map. Why?

Abstract (arXiv)

Contrastive learning (CL) has emerged as a powerful technique for representation learning, with or without label supervision. However, supervised CL is prone to collapsing representations of subclasses within a class by not capturing all their features, and unsupervised CL may suppress harder class-relevant features by focusing on learning easy class-irrelevant features; both significantly compromise representation quality. Yet, there is no theoretical understanding of textitclass collapse or textitfeature suppression at textittest time. We provide the first unified theoretically rigorous framework to determine textitwhich features are learnt by CL. Our analysis indicate that, perhaps surprisingly, bias of (stochastic) gradient descent towards finding simpler solutions is a key factor in collapsing subclass representations and suppressing harder class-relevant features. Moreover, we present increasing embedding dimensionality and improving the quality of data augmentations as two theoretically motivated solutions to feature suppression. We also provide the first theoretical explanation for why employing supervised and unsupervised CL together yields higher-quality representations, even when using commonly-used stochastic gradient methods.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 42 of its 59 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

42 of 59 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  4. 2020Supervised Contrastive Learningcited by 2,829
  5. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  6. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  8. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  9. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  10. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  11. 2020Debiased Contrastive Learningcited by 356
  12. 2017Implicit Regularization in Matrix Factorizationcited by 322
30 more
  1. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  2. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  3. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  4. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  5. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  6. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  7. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  8. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  9. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  10. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  11. 2020Intriguing Properties of Contrastive Lossescited by 130
  12. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 123
  13. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 119
  14. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  15. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  16. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  17. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  18. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
  19. 2021A Broad Study on the Transferability of Visual Representations with Contrastive Learningcited by 78
  20. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  21. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  22. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54
  23. 2021The Power of Contrast for Feature Learning: A Theoretical Analysiscited by 53
  24. 2016Convolutional Neural Networks Analyzed via Convolutional Sparse Codingcited by 52
  25. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
  26. 2014Sparse Modeling for Image and Vision Processingcited by 48
  27. 2022Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learningcited by 40
  28. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 40
  29. 2020Addressing Feature Suppression in Unsupervised Visual Representationscited by 13
  30. 2018Towards Understanding the Generalization Bias of Two Layer Convolutional Linear Classifiers with Gradient Descentcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 12; 2025: 12; 2026: 4.

30 papers.

Sort:

By area:

  1. 2024Learned feature representations are biased by complexity, learning order, position, and morecited by 23
  2. 2024Investigating the Benefits of Projection Head for Representation Learningcited by 20
  3. 2024Data-Efficient Contrastive Language-Image Pretraining: Prioritizing Data Quality over Quantitycited by 19
  4. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  5. 2024Complexity Matters: Dynamics of Feature Learning in the Presence of Spurious Correlationscited by 12
  6. 2024CLOSER: Towards Better Representation Learning for Few-Shot Class-Incremental Learningcited by 11
  7. 2025Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictionscited by 10
  8. 2024Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learningcited by 9
  9. 2025Leveraging Multilingual Training for Authorship Representation: Enhancing Generalization across Languages and Domainscited by 8
  10. 2024Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networkscited by 6
  11. 2025A Tale of Two Classes: Adapting Supervised Contrastive Learning to Binary Imbalanced Datasetscited by 5
  12. 2023Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shiftcited by 4
18 more
  1. 2024On the Effectiveness of Supervision in Asymmetric Non-Contrastive Learningcited by 3
  2. 2026Muon Learns More Robust and Transferable Features than Adamcited by 3
  3. 2023RetriBooru: Leakage-Free Retrieval of Conditions from Reference Images for Subject-Driven Generationcited by 2
  4. 2024Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP)cited by 2
  5. 2024Neural Collapse Inspired Knowledge Distillationcited by 2
  6. 2025A Theoretical Framework for Preventing Class Collapse in Supervised Contrastive Learningcited by 2
  7. 2025When and Where do Data Poisons Attack Textual Inversion?cited by 2
  8. 2024Exploiting Supervised Poison Vulnerability to Strengthen Self-Supervised Defensecited by 1
  9. 2025DivIL: Unveiling and Addressing Over-Invariance for Out-of- Distribution Generalizationcited by 1
  10. 2025BCE vs. CE in Deep Feature Learningcited by 1
  11. 2025A Critical Review of Predominant Bias in Neural Networkscited by 0
  12. 2025Informed Mixing -- Improving Open Set Recognition via Attribution-based Augmentationcited by 0
  13. 2025rETF-semiSL: Semi-Supervised Learning for Neural Collapse in Temporal Datacited by 0
  14. 2025Learning Without Augmenting: Unsupervised Time Series Representation Learning via Frame Projectionscited by 0
  15. 2025Semi-Supervised Contrastive Learning with Orthonormal Prototypescited by 0
  16. 2026TRACER: Persistent Regularization for Robust Multimodal Finetuningcited by 0
  17. 2026OSCS-SupCon: Orthogonal Sigmoid-based Common and Style Supervised Contrastive Learning for Robust Feature Disentanglementcited by 0
  18. 2026The Layer Mystery of VLA: An Information-Theoretical Analysis of VLA Latent Interfacecited by 0