mlsanityData to 7 Oct 2026

Contrasting Contrastive Self-Supervised Representation Learning Pipelines

arXiv 2103.14005 ↗First posted 25 Mar 2021cs.CV cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

In the past few years, we have witnessed remarkable breakthroughs in self-supervised representation learning. Despite the success and adoption of representations learned through this paradigm, much is yet to be understood about how different training methods and datasets influence performance on downstream tasks. In this paper, we analyze contrastive approaches as one of the most successful and popular variants of self-supervised representation learning. We perform this analysis from the perspective of the training algorithms, pre-training datasets and end tasks. We examine over 700 training experiments including 30 encoders, 4 pre-training datasets and 20 diverse downstream tasks. Our experiments address various questions regarding the performance of self-supervised models compared to their supervised counterparts, current benchmarks used for evaluation, and the effect of the pre-training data on end task performance. Our Visual Representation Benchmark (ViRB) is available at: https://github.com/allenai/virb.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 51 of its 67 references that are in the map. First posted on arXiv 25 Mar 2021.

Builds on

51 of 67 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  10. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  11. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  12. 2020Exploring Simple Siamese Representation Learningcited by 2,853
38 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  2. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  3. 2013Describing Textures in the Wildcited by 2,110
  4. 2017The Kinetics Human Action Video Datasetcited by 1,894
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  6. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  7. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  9. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  10. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  11. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  12. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  13. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  14. 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Networkcited by 1,199
  15. 2016Colorful Image Colorizationcited by 1,113
  16. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  17. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  18. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  19. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  20. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  21. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  22. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  23. 2018Do Better ImageNet Models Transfer Better?cited by 560
  24. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  25. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  26. 2016Adversarial Feature Learningcited by 350
  27. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  28. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  29. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  30. 2020Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biasescited by 143
  31. 2016Learning Features by Watching Objects Movecited by 138
  32. 2017Representation Learning by Learning to Countcited by 120
  33. 2020What makes instance discrimination good for transfer learning?cited by 111
  34. 2015Learning to See by Movingcited by 101
  35. 2021Self-Supervised Pretraining Improves Self-Supervised Pretrainingcited by 81
  36. 2020How Useful is Self-Supervised Pretraining for Visual Tasks?cited by 80
  37. 2019A critical analysis of self-supervision, or what we can learn from a single imagecited by 75
  38. 2015Learning image representations tied to ego-motioncited by 41

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 7; 2022: 10; 2023: 16; 2024: 2; 2025: 2.

37 papers.

Sort:

  1. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  2. 2023CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoderscited by 167
  3. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 122
  4. 2021An Empirical Study of Graph Contrastive Learningcited by 113
  5. 2021Large-scale Unsupervised Semantic Segmentationcited by 98
  6. 2021When Does Contrastive Visual Representation Learning Work?cited by 92
  7. 2022Teaching Matters: Investigating the Role of Supervision in Vision Transformerscited by 45
  8. 2022Dataset Inference for Self-Supervised Modelscited by 36
  9. 2023Does progress on ImageNet transfer to real-world datasets?cited by 35
  10. 2021Unsupervised Domain Generalization by Learning a Bridge Across Domainscited by 33
  11. 2023The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learningcited by 27
  12. 2023TrojText: Test-time Invisible Textual Trojan Insertioncited by 26
25 more
  1. 2022How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?cited by 23
  2. 2022Consecutive Pretraining: A Knowledge Transfer Learning Strategy with Relevant Unlabeled Data for Remote Sensing Domaincited by 22
  3. 2022Beyond Supervised vs. Unsupervised: Representative Benchmarking and Analysis of Image Representation Learningcited by 20
  4. 2021Learning to Align Sequential Actions in the Wildcited by 19
  5. 2023Simplifying Neural Network Training Under Class Imbalancecited by 18
  6. 2022Improving Fine-tuning of Self-supervised Models with Contrastive Initializationcited by 10
  7. 2023A Review of Predictive and Contrastive Self-supervised Learning for Medical Imagescited by 10
  8. 2023A Holistic Assessment of the Reliability of Machine Learning Systemscited by 9
  9. 2022Measuring the Interpretability of Unsupervised Representations via Quantized Reverse Probingcited by 8
  10. 2023Task-Oriented Koopman-Based Control with Contrastive Encodercited by 8
  11. 2023Scale-Adaptive Feature Aggregation for Efficient Space-Time Video Super-Resolutioncited by 8
  12. 2023Rethinking Self-Supervised Visual Representation Learning in Pre-training for 3D Human Pose and Shape Estimationcited by 7
  13. 2024On Pretraining Data Diversity for Self-Supervised Learningcited by 7
  14. 2022Empirical Evaluation and Theoretical Analysis for Representation Learning: A Surveycited by 5
  15. 2023Can Pre-trained Networks Detect Familiar Out-of-Distribution Data?cited by 4
  16. 2025SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learningcited by 4
  17. 2022Pixel-level Correspondence for Self-Supervised Learning from Videocited by 3
  18. 2023Constructive Assimilation: Boosting Contrastive Learning Performance through View Generation Strategiescited by 3
  19. 2025Self-Supervised Learning for Image Segmentation: A Comprehensive Surveycited by 3
  20. 2021Tradeoffs Between Contrastive and Supervised Learning: An Empirical Studycited by 2
  21. 2023What Constitutes Good Contrastive Learning in Time-Series Forecasting?cited by 2
  22. 2022Simple Control Baselines for Evaluating Transfer Learningcited by 1
  23. 2023Patch-Wise Self-Supervised Visual Representation Learning: A Fine-Grained Approachcited by 1
  24. 2024Pre-training for Action Recognition with Automatically Generated Fractal Datasetscited by 1
  25. 2023Multi-entity Video Transformers for Fine-Grained Video Representation Learningcited by 0