mlsanityData to 7 Oct 2026

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

arXiv 2103.14030 ↗First posted 25 Mar 2021cs.CV cs.LGcited by 12,082

Not on the map. Why?

Cited by 407 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with Shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures. The code and models are publicly available at~urlhttps://github.com/microsoft/Swin-Transformer.

Key facts. Cited by 12,082 papers in the map (data through 7 Oct 2026). Builds on 71 of its 86 references that are in the map. First posted on arXiv 25 Mar 2021.

Builds on

71 of 86 references.

References · 71

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2017Mask R-CNNcited by 7,259
  12. 2016Densely Connected Convolutional Networkscited by 6,691
38 more
  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2014Going Deeper with Convolutionscited by 4,231
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  8. 2016Wide Residual Networkscited by 3,057
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  10. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  11. 2017Non-local Neural Networkscited by 2,290
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  13. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  14. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  15. 2020Rethinking Attention with Performerscited by 1,584
  16. 2017Deformable Convolutional Networkscited by 1,481
  17. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  18. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  19. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  20. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  21. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  22. 2017Random Erasing Data Augmentationcited by 1,235
  23. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  24. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  25. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,018
  26. 2018Dual Attention Network for Scene Segmentationcited by 1,001
  27. 2016Deep Networks with Stochastic Depthcited by 973
  28. 2020Designing Network Design Spacescited by 954
  29. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  30. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  31. 2018Deformable ConvNets v2: More Deformable, Better Resultscited by 666
  32. 2021Transformer in Transformercited by 632
  33. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  34. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 573
  35. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  36. 2019Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selectioncited by 539
  37. 2020Sparse R-CNN: End-to-End Object Detection with Learnable Proposalscited by 494
  38. 2019Hybrid Task Cascade for Instance Segmentationcited by 479

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 539; 2022: 1935; 2023: 2828; 2024: 2786; 2025: 2471; 2026: 1523.

12,082 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022A ConvNet for the 2020scited by 3,545
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  4. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  7. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  8. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,869
  9. 2022Visual Prompt Tuningcited by 1,674
  10. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  11. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,618
  12. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
38 more
  1. 2024Depth Anything V2cited by 1,385
  2. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  3. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  4. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  5. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,218
  6. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  7. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  8. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  9. 2021Grounded Language-Image Pre-trainingcited by 1,147
  10. 2021Video Swin Transformercited by 1,090
  11. 2021YOLOX: Exceeding YOLO Series in 2021cited by 1,064
  12. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  13. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,030
  14. 2022Token Merging: Your ViT But Fastercited by 995
  15. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  16. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  17. 2024VMamba: Visual State Space Modelcited by 955
  18. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  19. 2021Multiscale Vision Transformerscited by 896
  20. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  21. 2021Uformer: A General U-Shaped Transformer for Image Restorationcited by 864
  22. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  23. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  24. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  25. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  26. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  27. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  28. 2022Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Imagescited by 823
  29. 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 812
  30. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  31. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  32. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  33. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  34. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  35. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  36. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  37. 2022Simple Baselines for Image Restorationcited by 701
  38. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646

50 most cited of 12,082 All 12,082 (JSON)