mlsanityData to 7 Oct 2026

Swin Transformer V2: Scaling Up Capacity and Resolution

arXiv 2111.09883 ↗First posted 18 Nov 2021cs.CVcited by 1,215

Not on the map. Why?

Cited by 31 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large-scale NLP models have been shown to significantly improve the performance on language tasks with no signs of saturation. They also demonstrate amazing few-shot capabilities like that of human beings. This paper aims to explore large-scale models in computer vision. We tackle three major issues in training and application of large vision models, including training instability, resolution gaps between pre-training and fine-tuning, and hunger on labelled data. Three main techniques are proposed: 1) a residual-post-norm method combined with cosine attention to improve training stability; 2) A log-spaced continuous position bias method to effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; 3) A self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images. Through these techniques, this paper successfully trained a 3 billion-parameter Swin Transformer V2 model, which is the largest dense vision model to date, and makes it capable of training with images of up to 1,536×1,536 resolution. It set new performance records on 4 representative vision tasks, including ImageNet-V2 image classification, COCO object detection, ADE20K semantic segmentation, and Kinetics-400 video action classification. Also note our training is much more efficient than that in Google's billion-level visual models, which consumes 40 times less labelled data and 40 times less training time. Code is available at urlhttps://github.com/microsoft/Swin-Transformer.

Key facts. Cited by 1,215 papers in the map (data through 7 Oct 2026). Builds on 74 of its 86 references that are in the map. First posted on arXiv 18 Nov 2021.

Builds on

74 of 86 references.

References · 74

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2020End-to-End Object Detection with Transformerscited by 7,384
  12. 2017Mask R-CNNcited by 7,259
38 more
  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2016Layer Normalizationcited by 5,579
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  11. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  12. 2017The Kinetics Human Action Video Datasetcited by 1,894
  13. 2021ViViT: A Video Vision Transformercited by 1,610
  14. 2018Group Normalizationcited by 1,514
  15. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  16. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  17. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  18. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  19. 2017Random Erasing Data Augmentationcited by 1,235
  20. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  21. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  22. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  23. 2021Video Swin Transformercited by 1,090
  24. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  25. 2021Scaling Vision Transformerscited by 993
  26. 2016Deep Networks with Stochastic Depthcited by 973
  27. 2020On Layer Normalization in the Transformer Architecturecited by 906
  28. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  29. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  30. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  31. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  32. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  33. 2017SchNet: A continuous-filter convolutional neural network for modeling quantum interactionscited by 569
  34. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  35. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  36. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  37. 2020Sparse R-CNN: End-to-End Object Detection with Learnable Proposalscited by 494
  38. 2019Hybrid Task Cascade for Instance Segmentationcited by 479

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 161; 2023: 326; 2024: 304; 2025: 250; 2026: 171.

1,215 papers.

Sort:

By area:

  1. 2024Depth Anything V2cited by 1,385
  2. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  3. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  4. 2022Token Merging: Your ViT But Fastercited by 995
  5. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  6. 2024VMamba: Visual State Space Modelcited by 955
  7. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  8. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  9. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  10. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  11. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  12. 2022Vision Transformer Adapter for Dense Predictionscited by 564
38 more
  1. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  2. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  3. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  4. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  5. 2024YOLOv9: Learning What You Want to Learn Using Programmable Gradient Informationcited by 422
  6. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 395
  7. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  8. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  9. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 341
  10. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  11. 2023ClimaX: A foundation model for weather and climatecited by 325
  12. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  13. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  14. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  15. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  16. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  17. 2022DETRs with Collaborative Hybrid Assignments Trainingcited by 273
  18. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
  19. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  20. 2023FuXi: A cascade machine learning forecasting system for 15-day global weather forecastcited by 237
  21. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  22. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  23. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  24. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  25. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  26. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  27. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  28. 2022Focal Modulation Networkscited by 186
  29. 2024Poseidon: Efficient Foundation Models for PDEscited by 186
  30. 2023RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Modelcited by 180
  31. 2022DKM: Dense Kernelized Feature Matching for Geometry Estimationcited by 174
  32. 2023MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimationcited by 173
  33. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  34. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  35. 2022CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flowcited by 165
  36. 2023MedNeXt: Transformer-driven Scaling of ConvNets for Medical Image Segmentationcited by 161
  37. 2022Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNscited by 155
  38. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154

50 most cited of 1,215 All 1,215 (JSON)