mlsanityData to 7 Oct 2026

Supervised Contrastive Learning

arXiv 2004.11362 ↗First posted 23 Apr 2020cs.LG cs.CV stat.MLcited by 2,803

Not on the map. Why?

Cited by 41 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive learning applied to self-supervised representation learning has seen a resurgence in recent years, leading to state of the art performance in the unsupervised training of deep image models. Modern batch contrastive approaches subsume or significantly outperform traditional contrastive losses such as triplet, max-margin and the N-pairs loss. In this work, we extend the self-supervised batch contrastive approach to the fully-supervised setting, allowing us to effectively leverage label information. Clusters of points belonging to the same class are pulled together in embedding space, while simultaneously pushing apart clusters of samples from different classes. We analyze two possible versions of the supervised contrastive (SupCon) loss, identifying the best-performing formulation of the loss. On ResNet-200, we achieve top-1 accuracy of 81.4% on the ImageNet dataset, which is 0.8% above the best number reported for this architecture. We show consistent outperformance over cross-entropy on other datasets and two ResNet variants. The loss shows benefits for robustness to natural corruptions and is more stable to hyperparameter settings such as optimizers and data augmentations. Our loss function is simple to implement, and reference TensorFlow code is released at https://t.ly/supcon.

Key facts. Cited by 2,803 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 342; 2022: 560; 2023: 593; 2024: 449; 2025: 446; 2026: 413.

2,803 papers.

Sort:

By area:

  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  3. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
  4. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  5. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  6. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  7. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  8. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  9. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  10. 2021SSD: A Unified Framework for Self-Supervised Outlier Detectioncited by 264
  11. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  12. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
38 more
  1. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  2. 2021Co2L: Contrastive Continual Learningcited by 243
  3. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  4. 2021Exploring Cross-Image Pixel Contrast for Semantic Segmentationcited by 242
  5. 2021Parametric Contrastive Learningcited by 217
  6. 2021New Insights on Reducing Abrupt Representation Change in Online Continual Learningcited by 215
  7. 2022Generalized Category Discoverycited by 209
  8. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  9. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  10. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  11. 2022MotionBERT: A Unified Perspective on Learning Human Motion Representationscited by 188
  12. 2021Self-Supervised Learning of Graph Neural Networks: A Unified Reviewcited by 184
  13. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  14. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
  15. 2021Learning to Diversify for Single Domain Generalizationcited by 170
  16. 2021FSCE: Few-Shot Object Detection via Contrastive Proposal Encodingcited by 168
  17. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  18. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 162
  19. 2021A Unified Objective for Novel Class Discoverycited by 161
  20. 2021A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challengescited by 154
  21. 2021Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Datacited by 152
  22. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 152
  23. 2021Self-Supervised Models are Continual Learnerscited by 149
  24. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  25. 2022Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphscited by 144
  26. 2022Balanced Contrastive Learning for Long-Tailed Visual Recognitioncited by 143
  27. 2021Decoupled Contrastive Learningcited by 141
  28. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 141
  29. 2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 140
  30. 2021Targeted Supervised Contrastive Learning for Long-Tailed Recognitioncited by 132
  31. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  32. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
  33. 2021Contrastive Learning based Hybrid Networks for Long-Tailed Image Classificationcited by 130
  34. 2022TruFor: Leveraging all-round clues for trustworthy image forgery detection and localizationcited by 130
  35. 2022How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?cited by 128
  36. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  37. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 122
  38. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 121

50 most cited of 2,803 All 2,803 (JSON)