mlsanityData to 7 Oct 2026

A Survey of the Self Supervised Learning Mechanisms for Vision Transformers

arXiv 2408.17059 ↗First posted 30 Aug 2024cs.CV cs.AI cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Advances in deep learning are re-defining how visual data is processed and understand by the machines. Vision Transformers (ViTs) have recently demonstrated prominent performance in computer vision related tasks. However, their performance improves with increasing numbers of labeled data, indicating reliance on labeled data. Humanly annotated data are difficult to acquire and thus shifted the focus from traditional annotations to unsupervised learning strategies that learn structures inside the data. In response to this challenge, self-supervised learning (SSL) has emerged as a promising technique. SSL utilize inherent relationships within the data as a form of supervision. This technique can reduce the dependence on manual annotations and offers a more scalable and resource-effective approach to training models. Taking these strengths into account, it is necessary to assess the combination of SSL methods with ViTs, especially in the cases of limited labeled data. Inspired by this evolving trend, this survey aims to systematically review SSL mechanisms tailored for ViTs. We propose a comprehensive taxonomy to classify SSL techniques based on their representations and pre-training tasks. Furthermore, we highlighted the motivations behind the study of SSL, reviewed prominent pre-training tasks, and highlight advancements and challenges in this field. Furthermore, we conduct a comparative analysis of various SSL methods designed for ViTs, evaluating their strengths, limitations, and applicability to different scenarios.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 121 of its 198 references that are in the map. First posted on arXiv 30 Aug 2024.

Builds on

121 of 198 references.

References · 121

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  11. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  12. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  5. 2022Generative Adversarial Networkscited by 5,603
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  11. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  13. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  14. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  15. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  16. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  17. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  18. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  19. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  20. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  21. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  22. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  23. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  24. 2015Spatial Transformer Networkscited by 1,423
  25. 2019Natural Adversarial Examplescited by 1,401
  26. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  27. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  28. 2019Contrastive Multiview Codingcited by 1,297
  29. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  30. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  31. 2014How transferable are features in deep neural networks?cited by 1,216
  32. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  33. 2016Colorful Image Colorizationcited by 1,113
  34. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  35. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  36. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  37. 2019An Introduction to Variational Autoencoderscited by 997
  38. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982

50 most cited of 121 All 121 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 1.

8 papers.

  1. 2025A Survey on Remote Sensing Foundation Models: From Vision to Multimodalitycited by 18
  2. 2024SatVision-TOA: A Geospatial Foundation Model for Coarse-Resolution All-Sky Remote Sensing Imagerycited by 7
  3. 2024Multi-objective Deep Learning: Taxonomy and Survey of the State of the Artcited by 6
  4. 2025Survey of Multimodal Geospatial Foundation Models: Techniques, Applications, and Challengescited by 3
  5. 2025AutoGen Driven Multi Agent Framework for Iterative Crime Data Analysis and Predictioncited by 2
  6. 2025Automated MRI Tumor Segmentation using hybrid U-Net with Transformer and Efficient Attentioncited by 0
  7. 2025HMSViT: A Hierarchical Masked Self-Supervised Vision Transformer for Corneal Nerve Segmentation and Diabetic Neuropathy Diagnosiscited by 0
  8. 2026Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Modelscited by 0