mlsanityData to 7 Oct 2026

Self-supervised Learning: Generative or Contrastive

arXiv 2006.08218 ↗First posted 15 Jun 2020cs.LG stat.MLcited by 693

Not on the map. Why?

Abstract (arXiv)

Deep supervised learning has achieved great success in the last decade. However, its deficiencies of dependence on manual labels and vulnerability to attacks have driven people to explore a better solution. As an alternative, self-supervised learning attracts many researchers for its soaring performance on representation learning in the last several years. Self-supervised representation learning leverages input data itself as supervision and benefits almost all types of downstream tasks. In this survey, we take a look into new self-supervised learning methods for representation in computer vision, natural language processing, and graph learning. We comprehensively review the existing empirical methods and summarize them into three main categories according to their objectives: generative, contrastive, and generative-contrastive (adversarial). We further investigate related theoretical analysis work to provide deeper thoughts on how self-supervised learning works. Finally, we briefly discuss open problems and future directions for self-supervised learning. An outline slide for the survey is provided.

Key facts. Cited by 693 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 102; 2022: 162; 2023: 168; 2024: 130; 2025: 93; 2026: 38.

693 papers.

Sort:

By area:

  1. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  2. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  3. 2021Transformers in Vision: A Surveycited by 1,009
  4. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  5. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  6. 2021GPT Understands, Toocited by 715
  7. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  8. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  9. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  10. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  11. 2022GraphMAE: Self-Supervised Masked Graph Autoencoderscited by 443
  12. 2021Pre-Trained Models: Past, Present and Futurecited by 319
38 more
  1. 2021Graph Self-Supervised Learning: A Surveycited by 289
  2. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  3. 2021A Survey on Deep Semi-supervised Learningcited by 250
  4. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  5. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  6. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  7. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  8. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  9. 2024Deep Time Series Models: A Comprehensive Survey and Benchmarkcited by 191
  10. 2024CBraMod: A Criss-Cross Brain Foundation Model for EEG Decodingcited by 188
  11. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  12. 2021Self-Supervised Learning of Graph Neural Networks: A Unified Reviewcited by 184
  13. 2021Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learningcited by 175
  14. 2022Backdoor Defense via Decoupling the Training Processcited by 149
  15. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 141
  16. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  17. 2021Self-supervised Learning on Graphs: Contrastive, Generative,or Predictivecited by 136
  18. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  19. 2023GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learnercited by 131
  20. 2021Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learningcited by 130
  21. 2023SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modelingcited by 112
  22. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  23. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  24. 2023WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferencescited by 100
  25. 2023Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospectscited by 97
  26. 2021Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Seriescited by 96
  27. 2024AutoGLM: Autonomous Foundation Agents for GUIscited by 96
  28. 2021Augmentation-Free Self-Supervised Learning on Graphscited by 94
  29. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  30. 2022Intent Contrastive Learning for Sequential Recommendationcited by 90
  31. 2021Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrastcited by 85
  32. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  33. 2022SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observationcited by 82
  34. 2021Self-Supervised Pretraining Improves Self-Supervised Pretrainingcited by 81
  35. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  36. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  37. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  38. 2021Self-supervised learning methods and applications in medical imaging analysis: A surveycited by 76

50 most cited of 693 All 693 (JSON)