mlsanityData to 7 Oct 2026

When Does Contrastive Visual Representation Learning Work?

arXiv 2105.05837 ↗First posted 12 May 2021cs.CV cs.LGcited by 92

Not on the map. Why?

Abstract (arXiv)

Recent self-supervised representation learning techniques have largely closed the gap between supervised and unsupervised learning on ImageNet classification. While the particulars of pretraining on ImageNet are now relatively well understood, the field still lacks widely accepted best practices for replicating this success on other datasets. As a first step in this direction, we study contrastive self-supervised learning on four diverse large-scale datasets. By looking through the lenses of data quantity, data domain, data quality, and task granularity, we provide new insights into the necessary conditions for successful self-supervised learning. Our key findings include observations such as: (i) the benefit of additional pretraining data beyond 500k images is modest, (ii) adding pretraining images from another domain does not lead to more general representations, (iii) corrupted pretraining images have a disparate impact on supervised and self-supervised pretraining, and (iv) contrastive learning lags far behind supervised learning on fine-grained visual classification tasks.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Builds on 57 of its 69 references that are in the map. First posted on arXiv 12 May 2021.

Builds on

57 of 69 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  12. 2020Supervised Contrastive Learningcited by 2,803
38 more
  1. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  2. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  3. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  4. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  5. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  6. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  7. 2019Contrastive Multiview Codingcited by 1,297
  8. 2016Colorful Image Colorizationcited by 1,113
  9. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  10. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  11. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  12. 2018Recognition in Terra Incognitacited by 655
  13. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  14. 2017Large Batch Training of Convolutional Networkscited by 529
  15. 2020Rethinking Pre-training and Self-trainingcited by 352
  16. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  17. 2021Exploring Cross-Image Pixel Contrast for Semantic Segmentationcited by 242
  18. 2015YFCC100M: The New Data in Multimedia Researchcited by 234
  19. 2020What Should Not Be Contrastive in Contrastive Learningcited by 224
  20. 2017Unsupervised Representation Learning by Sorting Sequencescited by 219
  21. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  22. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  23. 2019Fixing the train-test resolution discrepancycited by 209
  24. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  25. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  26. 2020Geography-Aware Self-Supervised Learningcited by 190
  27. 2020Contrastive Learning of General-Purpose Audio Representationscited by 168
  28. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 167
  29. 2021Efficient Visual Pretraining with Contrastive Detectioncited by 150
  30. 2020Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biasescited by 143
  31. 2020Intriguing Properties of Contrastive Lossescited by 129
  32. 2020What makes instance discrimination good for transfer learning?cited by 111
  33. 2017Critical Learning Periods in Deep Neural Networkscited by 78
  34. 2020MoCo-CXR: MoCo Pretraining Improves Representation and Transferability of Chest X-ray Modelscited by 61
  35. 2020Are all negatives created equal in contrastive instance discrimination?cited by 50
  36. 2020Parametric Instance Classification for Unsupervised Visual Feature Learningcited by 41
  37. 2021Contrasting Contrastive Self-Supervised Representation Learning Pipelinescited by 37
  38. 2020Concept Generalization in Visual Representation Learningcited by 35

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 31; 2023: 21; 2024: 17; 2025: 6; 2026: 3.

92 papers.

Sort:

By area:

  1. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  2. 2023BioCLIP: A Vision Foundation Model for the Tree of Lifecited by 159
  3. 2021Fine-Grained Image Analysis with Deep Learning: A Surveycited by 135
  4. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 122
  5. 2021An Empirical Study of Graph Contrastive Learningcited by 113
  6. 2021Large-scale Unsupervised Semantic Segmentationcited by 98
  7. 2021Extending the WILDS Benchmark for Unsupervised Adaptationcited by 85
  8. 2022SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observationcited by 82
  9. 2023Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lecturescited by 68
  10. 2022Assessing the State of Self-Supervised Human Activity Recognition using Wearablescited by 63
  11. 2022Teaching Matters: Investigating the Role of Supervision in Vision Transformerscited by 45
  12. 2021SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Drivingcited by 43
38 more
  1. 2022Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learningcited by 39
  2. 2022Understanding Cross-Domain Few-Shot Learning Based on Domain Similarity and Few-Shot Difficultycited by 37
  3. 2023DreamTeacher: Pretraining Image Backbones with Deep Generative Modelscited by 37
  4. 2022TOV: The Original Vision Model for Optical Remote Sensing Image Understanding via Self-supervised Learningcited by 35
  5. 2022SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Modelscited by 34
  6. 2024SelEx: Self-Expertise in Fine-Grained Generalized Category Discoverycited by 34
  7. 2023The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learningcited by 27
  8. 2022A simple, efficient and scalable contrastive masked autoencoder for learning visual representationscited by 25
  9. 2025Perch 2.0: The Bittern Lesson for Bioacousticscited by 25
  10. 2022Learning Gait Representation from Massive Unlabelled Walking Videos: A Benchmarkcited by 24
  11. 2022How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?cited by 23
  12. 2023MaskCon: Masked Contrastive Learning for Coarse-Labelled Datasetcited by 21
  13. 2022Beyond Supervised vs. Unsupervised: Representative Benchmarking and Analysis of Image Representation Learningcited by 20
  14. 2024A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classificationcited by 19
  15. 2022Task-Customized Self-Supervised Pre-training with Scalable Dynamic Routingcited by 18
  16. 2022Multimodal contrastive learning for remote sensing taskscited by 18
  17. 2023From Global to Local: Multi-scale Out-of-distribution Detectioncited by 18
  18. 2024CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scalecited by 18
  19. 2022Is Self-Supervised Learning More Robust Than Supervised Learning?cited by 17
  20. 2023GeoNet: Benchmarking Unsupervised Adaptation across Geographiescited by 17
  21. 2022Joint Self-Supervised Image-Volume Representation Learning with Intra-Inter Contrastive Clusteringcited by 16
  22. 2023(LC)2: LiDAR-Camera Loop Constraints For Cross-Modal Place Recognitioncited by 16
  23. 2023Self-Supervised Disentanglement by Leveraging Structure in Data Augmentationscited by 16
  24. 2022MemSAC: Memory Augmented Sample Consistency for Large Scale Unsupervised Domain Adaptationcited by 15
  25. 2023Learning Common Rationale to Improve Self-Supervised Representation for Fine-Grained Visual Recognition Problemscited by 14
  26. 2021Improving Self-supervised Learning with Hardness-aware Dynamic Curriculum Learning: An Application to Digital Pathologycited by 13
  27. 2021Learning Representations with Contrastive Self-Supervised Learning for Histopathology Applicationscited by 12
  28. 2022On Label Granularity and Object Localizationcited by 12
  29. 2023Coreset Sampling from Open-Set for Fine-Grained Self-Supervised Learningcited by 12
  30. 2021Semi-Supervised Learning with Taxonomic Labelscited by 11
  31. 2025Taxonomy-Aware Evaluation of Vision-Language Modelscited by 11
  32. 2021An Evaluation of Self-Supervised Pre-Training for Skin-Lesion Analysiscited by 10
  33. 2024Contrastive ground-level image and remote sensing pre-training improves representation learning for natural world imagerycited by 10
  34. 2022Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learningcited by 9
  35. 2024CaBuAr: California Burned Areas dataset for delineationcited by 9
  36. 2022Measuring the Interpretability of Unsupervised Representations via Quantized Reverse Probingcited by 8
  37. 2023Clinically Labeled Contrastive Learning for OCT Biomarker Classificationcited by 8
  38. 2021Using Self-Supervised Auxiliary Tasks to Improve Fine-Grained Facial Representationcited by 7

50 most cited of 92 All 92 (JSON)