mlsanityData to 7 Oct 2026

A Survey on Contrastive Self-supervised Learning

arXiv 2011.00362 ↗First posted 31 Oct 2020cs.CVcited by 553

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning has gained popularity because of its ability to avoid the cost of annotating large-scale datasets. It is capable of adopting self-defined pseudo labels as supervision and use the learned representations for several downstream tasks. Specifically, contrastive learning has recently become a dominant component in self-supervised learning methods for computer vision, natural language processing (NLP), and other domains. It aims at embedding augmented versions of the same sample close to each other while trying to push away embeddings from different samples. This paper provides an extensive review of self-supervised methods that follow the contrastive approach. The work explains commonly used pretext tasks in a contrastive learning setup, followed by different architectures that have been proposed so far. Next, we have a performance comparison of different methods for multiple downstream tasks such as image classification, object detection, and action recognition. Finally, we conclude with the limitations of the current methods and the need for further techniques and future directions to make substantial progress.

Key facts. Cited by 553 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 46; 2022: 109; 2023: 141; 2024: 145; 2025: 81; 2026: 31.

553 papers.

Sort:

By area:

  1. 2022SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagerycited by 402
  2. 2021Graph Self-Supervised Learning: A Surveycited by 289
  3. 2021A Survey on Deep Semi-supervised Learningcited by 250
  4. 2022Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistencycited by 246
  5. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  6. 2024Deep Time Series Models: A Comprehensive Survey and Benchmarkcited by 191
  7. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  8. 2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 140
  9. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  10. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  11. 2023SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modelingcited by 112
  12. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
38 more
  1. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  2. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  3. 2023GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretationcited by 101
  4. 2022Out-Of-Distribution Generalization on Graphs: A Surveycited by 97
  5. 2023Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospectscited by 97
  6. 2021Region Similarity Representation Learningcited by 96
  7. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  8. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  9. 2021Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrastcited by 85
  10. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  11. 2021Self-supervised learning methods and applications in medical imaging analysis: A surveycited by 76
  12. 2022A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directionscited by 73
  13. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  14. 2022Masked Image Modeling Advances 3D Medical Image Analysiscited by 71
  15. 2022Audio Self-supervised Learning: A Surveycited by 65
  16. 2022Model Reprogramming: Resource-Efficient Cross-Domain Machine Learningcited by 64
  17. 2021Are we ready for a new paradigm shift? A Survey on Visual Deep MLPcited by 63
  18. 2023Frozen Language Model Helps ECG Zero-Shot Learningcited by 62
  19. 2023A Billion-scale Foundation Model for Remote Sensing Imagescited by 62
  20. 2024GuardT2I: Defending Text-to-Image Models from Adversarial Promptscited by 61
  21. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  22. 2021Self-Supervised Learning of Remote Sensing Scene Representations Using Contrastive Multiview Codingcited by 58
  23. 2021Detecting Corrupted Labels Without Training a Model to Predictcited by 57
  24. 2021SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representationscited by 55
  25. 2022Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANscited by 55
  26. 2021The Power of Contrast for Feature Learning: A Theoretical Analysiscited by 53
  27. 2022Residual Local Feature Network for Efficient Super-Resolutioncited by 53
  28. 2022A Closer Look at Few-shot Image Generationcited by 49
  29. 2023CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understandingcited by 48
  30. 2021Deep Learning for HDR Imaging: State-of-the-Art and Future Trendscited by 45
  31. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  32. 2021A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectivescited by 41
  33. 2022AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoderscited by 41
  34. 2022Self-supervised Contrastive Representation Learning for Semi-supervised Time-Series Classificationcited by 39
  35. 2023Generalized Graph Prompt: Toward a Unification of Pre-Training and Downstream Tasks on Graphscited by 39
  36. 2021Chemical-Reaction-Aware Molecule Representation Learningcited by 36
  37. 2022Dataset Inference for Self-Supervised Modelscited by 36
  38. 2022Understanding and Improving the Role of Projection Head in Self-Supervised Learningcited by 36

50 most cited of 553 All 553 (JSON)