mlsanityData to 7 Oct 2026

Self-Supervised Learning with Swin Transformers

arXiv 2105.04553 ↗First posted 10 May 2021cs.CVcited by 121

Not on the map. Why?

Abstract (arXiv)

We are witnessing a modeling shift from CNN to Transformers in computer vision. In this work, we present a self-supervised learning approach called MoBY, with Vision Transformers as its backbone architecture. The approach basically has no new inventions, which is combined from MoCo v2 and BYOL and tuned to achieve reasonably high accuracy on ImageNet-1K linear evaluation: 72.8% and 75.0% top-1 accuracy using DeiT-S and Swin-T, respectively, by 300-epoch training. The performance is slightly better than recent works of MoCo v3 and DINO which adopt DeiT as the backbone, but with much lighter tricks. More importantly, the general-purpose Swin Transformer backbone enables us to also evaluate the learnt representations on downstream tasks such as object detection and semantic segmentation, in contrast to a few recent approaches built on ViT/DeiT which only report linear evaluation results on ImageNet-1K due to ViT/DeiT not tamed for these dense prediction tasks. We hope our results can facilitate more comprehensive evaluation of self-supervised learning methods designed for Transformer architectures. Our code and models are available at https://github.com/SwinTransformer/Transformer-SSL, which will be continually enriched.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Builds on 18 of its 19 references that are in the map. First posted on arXiv 10 May 2021.

Builds on

18 of 19 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2017Mask R-CNNcited by 7,259
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
6 more
  1. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  2. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  3. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  4. 2020Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learningcited by 297
  5. 2021Efficient Visual Pretraining with Contrastive Detectioncited by 150
  6. 2020Parametric Instance Classification for Unsupervised Visual Feature Learningcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 33; 2023: 30; 2024: 23; 2025: 13; 2026: 5.

121 papers.

Sort:

By area:

  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  2. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  3. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  4. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  5. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 418
  6. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  7. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  8. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  9. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  10. 2022Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNscited by 155
  11. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  12. 2024Mamba-UNet: UNet-Like Pure Visual Mamba for Medical Image Segmentationcited by 133
38 more
  1. 2021A Survey of Visual Transformerscited by 117
  2. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 117
  3. 2021MST: Masked Self-Supervised Transformer for Visual Representationcited by 114
  4. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
  5. 2021Ensembling Off-the-shelf Models for GAN Trainingcited by 84
  6. 2022Object discovery and representation networkscited by 80
  7. 2023Vision-Language Models as a Source of Rewardscited by 65
  8. 2022Patch-level Representation Learning for Self-supervised Vision Transformerscited by 60
  9. 2022Mugs: A Multi-Granular Self-Supervised Learning Frameworkcited by 54
  10. 2022Polyhistor: Parameter-Efficient Multi-Task Adaptation for Dense Vision Taskscited by 52
  11. 2022Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?cited by 47
  12. 2023FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Modelscited by 47
  13. 2021SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Drivingcited by 43
  14. 2022Modeling Image Composition for Complex Scene Generationcited by 43
  15. 2022Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motioncited by 42
  16. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  17. 2022Can CNNs Be More Robust Than Transformers?cited by 38
  18. 2021Self-Supervised Pre-Training for Transformer-Based Person Re-Identificationcited by 36
  19. 2021Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learningcited by 34
  20. 2022SimA: Simple Softmax-free Attention for Vision Transformerscited by 32
  21. 2022UniVIP: A Unified Framework for Self-Supervised Visual Pre-trainingcited by 30
  22. 2023ASSET: Robust Backdoor Data Detection Across a Multiplicity of Deep Learning Paradigmscited by 29
  23. 2022RePre: Improving Self-Supervised Vision Transformer with Reconstructive Pre-trainingcited by 24
  24. 2022Unsupervised Visual Representation Learning by Synchronous Momentum Groupingcited by 22
  25. 2022SupMAE: Supervised Masked Autoencoders Are Efficient Vision Learnerscited by 21
  26. 2022Beyond Supervised vs. Unsupervised: Representative Benchmarking and Analysis of Image Representation Learningcited by 20
  27. 2024NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation Learning for Neural Radiance Fieldscited by 20
  28. 2022Boosting vision transformers for image retrievalcited by 19
  29. 2023Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Modelcited by 19
  30. 2024Self-supervised Learning for Electroencephalogram: A Systematic Surveycited by 19
  31. 2022MimCo: Masked Image Modeling Pre-training with Contrastive Teachercited by 16
  32. 2022Attention Distillation: self-supervised vision transformer students need more guidancecited by 16
  33. 2022Object-wise Masked Autoencoders for Fast Pre-trainingcited by 15
  34. 2023Extended Agriculture-Vision: An Extension of a Large Aerial Image Dataset for Agricultural Pattern Analysiscited by 14
  35. 2023Patch-Level Contrasting without Patch Correspondence for Accurate and Dense Contrastive Representation Learningcited by 14
  36. 2024DINO as a von Mises-Fisher mixture modelcited by 14
  37. 2022SERE: Exploring Feature Self-relation for Self-supervised Transformercited by 12
  38. 2023Advancing Vision Transformers with Group-Mix Attentioncited by 11

50 most cited of 121 All 121 (JSON)