mlsanityData to 7 Oct 2026

Self-Supervised Representation Learning: Introduction, Advances and Challenges

arXiv 2110.09327 ↗First posted 18 Oct 2021cs.LG cs.CV stat.MLcited by 147

Not on the map. Why?

Abstract (arXiv)

Self-supervised representation learning methods aim to provide powerful deep feature learning without the requirement of large annotated datasets, thus alleviating the annotation bottleneck that is one of the main barriers to practical deployment of deep learning today. These methods have advanced rapidly in recent years, with their efficacy approaching and sometimes surpassing fully supervised pre-training alternatives across a variety of data modalities including image, video, sound, text and graphs. This article introduces this vibrant area including key concepts, the four main families of approach and associated state of the art, and how self-supervised methods are applied to diverse modalities of data. We further discuss practical considerations including workflows, representation transferability, and compute cost. Finally, we survey the major open challenges in the field that provide fertile ground for future work.

Key facts. Cited by 147 papers in the map (data through 7 Oct 2026). Builds on 78 of its 102 references that are in the map. First posted on arXiv 18 Oct 2021.

Builds on

78 of 102 references.

References · 78

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2017Inductive Representation Learning on Large Graphscited by 6,284
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
38 more
  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2016node2vec: Scalable Feature Learning for Networkscited by 1,913
  8. 2017The Kinetics Human Action Video Datasetcited by 1,894
  9. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  12. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  13. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  14. 2019Contrastive Multiview Codingcited by 1,297
  15. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  16. 2018Deep Graph Infomaxcited by 1,101
  17. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  18. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  19. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  20. 2019Strategies for Pre-training Graph Neural Networkscited by 964
  21. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  22. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  23. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  24. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  25. 2019Green AIcited by 620
  26. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  27. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  28. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  29. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  30. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  31. 2019InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximizationcited by 441
  32. 2020Debiased Contrastive Learningcited by 354
  33. 2020Rethinking Pre-training and Self-trainingcited by 352
  34. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  35. 2020GPT-GNN: Generative Pre-Training of Graph Neural Networkscited by 302
  36. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  37. 2018Deep Anomaly Detection Using Geometric Transformationscited by 269
  38. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 22; 2023: 52; 2024: 24; 2025: 34; 2026: 14.

147 papers.

Sort:

By area:

  1. 2022SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentationcited by 397
  2. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  3. 2022Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Differencecited by 123
  4. 2023Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospectscited by 97
  5. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 60
  6. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  7. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 57
  8. 2022Holo-Dex: Teaching Dexterity with Immersive Mixed Realitycited by 52
  9. 2023SELFormer: Molecular Representation Learning via SELFIES Language Modelscited by 40
  10. 2023Decoupling Common and Unique Representations for Multimodal Self-supervised Learningcited by 36
  11. 2023Large AI Model-Based Semantic Communicationscited by 35
  12. 2022Contrastive Audio-Language Learning for Musiccited by 32
38 more
  1. 2023Unpaired Multi-Domain Causal Representation Learningcited by 32
  2. 2022One-Shot Transfer of Affordance Regions? AffCorrs!cited by 31
  3. 2024Review of multimodal machine learning approaches in healthcarecited by 30
  4. 2025EEG Foundation Models: A Critical Review of Current Progress and Future Directionscited by 30
  5. 2024Universal Time-Series Representation Learning: A Surveycited by 28
  6. 2022Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Datacited by 27
  7. 2022Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Workscited by 24
  8. 2023ReConTab: Regularized Contrastive Representation Learning for Tabular Datacited by 24
  9. 2022Continual Barlow Twins: continual self-supervised learning for remote sensing semantic segmentationcited by 22
  10. 2022Consecutive Pretraining: A Knowledge Transfer Learning Strategy with Relevant Unlabeled Data for Remote Sensing Domaincited by 22
  11. 2023Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignalscited by 22
  12. 2022Controllable Data Generation by Deep Learning: A Reviewcited by 20
  13. 2022DDPM-CD: Denoising Diffusion Probabilistic Models as Feature Extractors for Change Detectioncited by 19
  14. 2023BIRB: A Generalization Benchmark for Information Retrieval in Bioacousticscited by 19
  15. 2023Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgettingcited by 16
  16. 2021Why Do Self-Supervised Models Transfer? Investigating the Impact of Invariance on Downstream Taskscited by 15
  17. 2025A Survey of Pathology Foundation Model: Progress and Future Directionscited by 14
  18. 2024Promoting cross-modal representations to improve multimodal foundation models for physiological signalscited by 13
  19. 2025Learning Clustering-based Prototypes for Compositional Zero-shot Learningcited by 13
  20. 2022Learning image representations for anomaly detection: application to discovery of histological alterations in drug developmentcited by 11
  21. 2024ExHuBERT: Enhancing HuBERT Through Block Extension and Fine-Tuning on 37 Emotion Datasetscited by 11
  22. 20256G WavesFM: A Foundation Model for Sensing, Communication, and Localizationcited by 11
  23. 2025Feed-Forward SceneDINO for Unsupervised Semantic Scene Completioncited by 11
  24. 2023A Review of Predictive and Contrastive Self-supervised Learning for Medical Imagescited by 10
  25. 2023A Generic Fundus Image Enhancement Network Boosted by Frequency Self-supervised Representation Learningcited by 10
  26. 2023CAD -- Contextual Multi-modal Alignment for Dynamic AVQAcited by 10
  27. 2025Scene-Centric Unsupervised Panoptic Segmentationcited by 10
  28. 2024Useful Blunders: Can Automated Speech Recognition Errors Improve Downstream Dementia Classification?cited by 9
  29. 2023Amortised Invariance Learning for Contrastive Self-Supervisioncited by 8
  30. 2024FEET: A Framework for Evaluating Embedding Techniquescited by 8
  31. 2023Unpaired Image-to-Image Translation with Limited Data to Reveal Subtle Phenotypescited by 7
  32. 2023Visual DNA: Representing and Comparing Images using Distributions of Neuron Activationscited by 7
  33. 2024ViLReF: An Expert Knowledge Enabled Vision-Language Retinal Foundation Modelcited by 7
  34. 2025MIRAGE: Multimodal foundation model and benchmark for comprehensive retinal OCT image analysiscited by 7
  35. 2025InfMasking: Unleashing Synergistic Information by Contrastive Multimodal Interactionscited by 7
  36. 2025Serialized EHR make for good text representationscited by 7
  37. 2025FNBench: Benchmarking Robust Federated Learning against Noisy Labelscited by 6
  38. 2022ProtoCLIP: Prototypical Contrastive Language Image Pretrainingcited by 5

50 most cited of 147 All 147 (JSON)