mlsanityData to 7 Oct 2026

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

arXiv 2104.13840 ↗First posted 28 Apr 2021cs.CV cs.AI cs.LGcited by 565

Not on the map. Why?

Abstract (arXiv)

Very recently, a variety of vision transformer architectures for dense prediction tasks have been proposed and they show that the design of spatial attention is critical to their success in these tasks. In this work, we revisit the design of the spatial attention and demonstrate that a carefully-devised yet simple spatial attention mechanism performs favourably against the state-of-the-art schemes. As a result, we propose two vision transformer architectures, namely, Twins-PCPVT and Twins-SVT. Our proposed architectures are highly-efficient and easy to implement, only involving matrix multiplications that are highly optimized in modern deep learning frameworks. More importantly, the proposed architectures achieve excellent performance on a wide range of visual tasks, including image level classification as well as dense detection and segmentation. The simplicity and strong performance suggest that our proposed architectures may serve as stronger backbones for many vision tasks. Our code is released at https://github.com/Meituan-AutoML/Twins .

Key facts. Cited by 565 papers in the map (data through 7 Oct 2026). Builds on 47 of its 53 references that are in the map. First posted on arXiv 28 Apr 2021.

Builds on

47 of 53 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2017Mask R-CNNcited by 7,259
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  11. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  12. 2017Squeeze-and-Excitation Networkscited by 4,763
35 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2014Going Deeper with Convolutionscited by 4,231
  3. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  5. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  7. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  8. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  9. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  10. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  11. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  12. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  13. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  14. 2016Deep Networks with Stochastic Depthcited by 973
  15. 2020Designing Network Design Spacescited by 954
  16. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  17. 2020Pre-Trained Image Processing Transformercited by 891
  18. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  19. 2018Image Transformercited by 770
  20. 2021Going deeper with Image Transformerscited by 759
  21. 2017Focal Loss for Dense Object Detectioncited by 749
  22. 2019Panoptic Feature Pyramid Networkscited by 644
  23. 2021Transformer in Transformercited by 632
  24. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  25. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  26. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  27. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  28. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 346
  29. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  30. 2017Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Networkcited by 208
  31. 2015ParseNet: Looking Wider to See Bettercited by 176
  32. 2020Learning Joint Spatial-Temporal Transformations for Video Inpaintingcited by 162
  33. 2018OCNet: Object Context Network for Scene Parsingcited by 140
  34. 2021BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Searchcited by 63
  35. 2014Rigid-Motion Scattering for Texture Classificationcited by 52

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 55; 2022: 184; 2023: 138; 2024: 103; 2025: 56; 2026: 29.

565 papers.

Sort:

By area:

  1. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  2. 2021Uformer: A General U-Shaped Transformer for Image Restorationcited by 864
  3. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  4. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  5. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  6. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  7. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  8. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  9. 2022Activating More Pixels in Image Super-Resolution Transformercited by 433
  10. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 395
  11. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  12. 2022V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformercited by 320
38 more
  1. 2022How Do Vision Transformers Work?cited by 305
  2. 2025YOLOv12: Attention-Centric Real-Time Object Detectorscited by 300
  3. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 263
  4. 2022Vision Transformers for Single Image Dehazingcited by 255
  5. 2024LocalMamba: Visual State Space Model with Windowed Selective Scancited by 235
  6. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 229
  7. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  8. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  9. 2022Visual Attention Networkcited by 210
  10. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  11. 2022Stratified Transformer for 3D Point Cloud Segmentationcited by 166
  12. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 161
  13. 2021Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformerscited by 159
  14. 2022Recurrent Vision Transformers for Object Detection with Event Camerascited by 158
  15. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  16. 2023BiFormer: Vision Transformer with Bi-Level Routing Attentioncited by 150
  17. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  18. 2024MambaOut: Do We Really Need Mamba for Vision?cited by 142
  19. 2022Stripformer: Strip Transformer for Fast Image Deblurringcited by 139
  20. 2022Multi-class Token Transformer for Weakly Supervised Semantic Segmentationcited by 138
  21. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  22. 2021SwinTrack: A Simple and Strong Baseline for Transformer Trackingcited by 126
  23. 2023Spherical Transformer for LiDAR-based 3D Recognitioncited by 124
  24. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  25. 2021MPViT: Multi-Path Vision Transformer for Dense Predictioncited by 122
  26. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  27. 2021A Survey of Visual Transformerscited by 117
  28. 2022ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyondcited by 117
  29. 2022EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformerscited by 116
  30. 2021Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformercited by 115
  31. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  32. 2023Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understandingcited by 111
  33. 2021Beyond Self-attention: External Attention using Two Linear Layers for Visual Taskscited by 109
  34. 2024DiTFastAttn: Attention Compression for Diffusion Transformer Modelscited by 109
  35. 2022SegViT: Semantic Segmentation with Plain Vision Transformerscited by 105
  36. 2023OctFormer: Octree-based Transformers for 3D Point Cloudscited by 105
  37. 2021MISSFormer: An Effective Medical Image Segmentation Transformercited by 103
  38. 2022HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutionscited by 103

50 most cited of 565 All 565 (JSON)