mlsanityData to 7 Oct 2026

Can contrastive learning avoid shortcut solutions?

arXiv 2106.11230 ↗First posted 21 Jun 2021cs.LGcited by 114

Not on the map. Why?

Abstract (arXiv)

The generalization of representations learned via contrastive learning depends crucially on what features of the data are extracted. However, we observe that the contrastive loss does not always sufficiently guide which features are extracted, a behavior that can negatively impact the performance on downstream tasks via "shortcuts", i.e., by inadvertently suppressing important predictive features. We find that feature extraction is influenced by the difficulty of the so-called instance discrimination task (i.e., the task of discriminating pairs of similar points from pairs of dissimilar ones). Although harder pairs improve the representation of some features, the improvement comes at the cost of suppressing previously well represented features. In response, we propose implicit feature modification (IFM), a method for altering positive and negative samples in order to guide contrastive models towards capturing a wider variety of predictive features. Empirically, we observe that IFM reduces feature suppression, and as a result improves performance on vision and medical imaging tasks. The code is available at: urlhttps://github.com/joshr17/IFM.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Builds on 51 of its 56 references that are in the map. First posted on arXiv 21 Jun 2021.

Builds on

51 of 56 references.

References · 51

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  8. 2013Intriguing properties of neural networkscited by 5,114
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
38 more
  1. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  2. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  3. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  5. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  6. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  7. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  8. 2019Contrastive Multiview Codingcited by 1,297
  9. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  10. 2016Colorful Image Colorizationcited by 1,113
  11. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  12. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  13. 2018Robustness May Be at Odds with Accuracycited by 873
  14. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  15. 2018Recognition in Terra Incognitacited by 655
  16. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  17. 2020Contrastive Learning with Hard Negative Samplescited by 619
  18. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  19. 2020Understanding the Behaviour of Contrastive Losscited by 397
  20. 2020Debiased Contrastive Learningcited by 354
  21. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  22. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  23. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  24. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  25. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  26. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  27. 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
  28. 2020Adversarial Self-Supervised Contrastive Learningcited by 138
  29. 2020Intriguing Properties of Contrastive Lossescited by 129
  30. 2020What shapes feature representations? Exploring datasets, architectures, and trainingcited by 125
  31. 2020What makes instance discrimination good for transfer learning?cited by 111
  32. 2021Contrastive Learning with Stronger Augmentationscited by 100
  33. 2020Contrastive Learning with Adversarial Examplescited by 91
  34. 2020AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversariescited by 90
  35. 2018Excessive Invariance Causes Adversarial Vulnerabilitycited by 66
  36. 2020Automatic Shortcut Removal for Self-Supervised Representation Learningcited by 43
  37. 2020Strength from Weakness: Fast Learning Using Weak Supervisioncited by 18
  38. 2020Unsupervised Representation Learning by InvariancePropagationcited by 18

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 32; 2023: 26; 2024: 23; 2025: 19; 2026: 12.

114 papers.

Sort:

By area:

  1. 2022Transformers Learn Shortcuts to Automatacited by 260
  2. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  3. 2023Improving CLIP Training with Language Rewritescited by 216
  4. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  5. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  6. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  7. 2023On the Foundations of Shortcut Learningcited by 64
  8. 2022Robust Contrastive Learning against Noisy Viewscited by 45
  9. 2023Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Datacited by 44
  10. 2021Robust Contrastive Learning Using Negative Samples with Diminished Semanticscited by 43
  11. 2022SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samplescited by 39
  12. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 39
38 more
  1. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  2. 2023ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classificationcited by 33
  3. 2023Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppressioncited by 30
  4. 2022Procedural Image Programs for Representation Learningcited by 28
  5. 2023What do neural networks learn in image classification? A frequency shortcut perspectivecited by 28
  6. 2023Too Large; Data Reduction for Vision-Language Pre-Trainingcited by 26
  7. 2022A simple, efficient and scalable contrastive masked autoencoder for learning visual representationscited by 25
  8. 2021Augmentations in Graph Contrastive Learning: Current Methodological Flaws & Towards Better Practicescited by 24
  9. 2022How Robust is Unsupervised Representation Learning to Distribution Shift?cited by 24
  10. 2022Joint Embedding Predictive Architectures Focus on Slow Featurescited by 24
  11. 2024Your Diffusion Model is Secretly a Certifiably Robust Classifiercited by 24
  12. 2024The Clever Hans Effect in Unsupervised Learningcited by 24
  13. 2022EquiMod: An Equivariance Module to Improve Self-Supervised Learningcited by 20
  14. 2023A Survey on the Robustness of Computer Vision Models against Common Corruptionscited by 20
  15. 2023From Hope to Safety: Unlearning Biases of Deep Models via Gradient Penalization in Latent Spacecited by 20
  16. 2022What shapes the loss landscape of self-supervised learning?cited by 19
  17. 2024HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolationcited by 17
  18. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  19. 2022A Contrastive Objective for Learning Disentangled Representationscited by 14
  20. 2022Rectify ViT Shortcut Learning by Visual Saliencycited by 14
  21. 2023Composition-contrastive Learning for Sentence Embeddingscited by 14
  22. 2024Demonstrating and Reducing Shortcuts in Vision-Language Representation Learningcited by 14
  23. 2024animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacousticscited by 14
  24. 2024Navigate Beyond Shortcuts: Debiased Learning through the Lens of Neural Collapsecited by 13
  25. 2022Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision?cited by 12
  26. 2022Integrating Prior Knowledge in Contrastive Learning with Kernelcited by 11
  27. 2022Improving Fine-tuning of Self-supervised Models with Contrastive Initializationcited by 10
  28. 2022CaCo: Both Positive and Negative Samples are Directly Learnable via Cooperative-adversarial Contrastive Learningcited by 9
  29. 2022Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learningcited by 9
  30. 2023ACTION++: Improving Semi-supervised Medical Image Segmentation with Adaptive Anatomical Contrastcited by 9
  31. 2024Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learningcited by 9
  32. 2022SDA: Simple Discrete Augmentation for Contrastive Sentence Representation Learningcited by 8
  33. 2022Self-supervised debiasing using low rank regularizationcited by 8
  34. 2023MAST: Masked Augmentation Subspace Training for Generalizable Self-Supervised Priorscited by 8
  35. 2024Understanding the Role of Equivariance in Self-supervised Learningcited by 7
  36. 2025Leveraging Multilingual Training for Authorship Representation: Enhancing Generalization across Languages and Domainscited by 7
  37. 2023Augmenting Chest X-ray Datasets with Non-Expert Annotationscited by 6
  38. 2023Removing Biases from Molecular Representations via Information Maximizationcited by 6

50 most cited of 114 All 114 (JSON)