mlsanityData to 7 Oct 2026

ScalableViT: Rethinking the Context-oriented Generalization of Vision Transformer

arXiv 2203.10790 ↗First posted 21 Mar 2022cs.CV cs.AIcited by 36

Not on the map. Why?

Abstract (arXiv)

The vanilla self-attention mechanism inherently relies on pre-defined and steadfast computational dimensions. Such inflexibility restricts it from possessing context-oriented generalization that can bring more contextual cues and global representations. To mitigate this issue, we propose a Scalable Self-Attention (SSA) mechanism that leverages two scaling factors to release dimensions of query, key, and value matrices while unbinding them with the input. This scalability fetches context-oriented generalization and enhances object sensitivity, which pushes the whole network into a more effective trade-off state between accuracy and cost. Furthermore, we propose an Interactive Window-based Self-Attention (IWSA), which establishes interaction between non-overlapping regions by re-merging independent value tokens and aggregating spatial information from adjacent windows. By stacking the SSA and IWSA alternately, the Scalable Vision Transformer (ScalableViT) achieves state-of-the-art performance in general-purpose vision tasks. For example, ScalableViT-S outperforms Twins-SVT-S by 1.4% and Swin-T by 1.8% on ImageNet-1K classification.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 48 of its 54 references that are in the map. First posted on arXiv 21 Mar 2022.

Builds on

48 of 54 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2017Mask R-CNNcited by 7,259
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  12. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
36 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2014Going Deeper with Convolutionscited by 4,231
  3. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  4. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  5. 2017Non-local Neural Networkscited by 2,290
  6. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  7. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  8. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  9. 2017Random Erasing Data Augmentationcited by 1,235
  10. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  11. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  12. 2016Deep Networks with Stochastic Depthcited by 973
  13. 2020Designing Network Design Spacescited by 954
  14. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  15. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  16. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  17. 2019Panoptic Feature Pyramid Networkscited by 644
  18. 2021Transformer in Transformercited by 632
  19. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  20. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  21. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  22. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  23. 2019Attention Augmented Convolutional Networkscited by 363
  24. 2019GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyondcited by 331
  25. 2020Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentationcited by 326
  26. 2020Exploring Self-attention for Image Recognitioncited by 313
  27. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  28. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  29. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  30. 2019Local Relation Networks for Image Recognitioncited by 210
  31. 2020How Much Position Information Do Convolutional Neural Networks Encode?cited by 171
  32. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  33. 2021CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attentioncited by 139
  34. 2020Disentangled Non-Local Neural Networkscited by 132
  35. 2021Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformercited by 115
  36. 2019Interlaced Sparse Self-Attention for Semantic Segmentationcited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 20; 2024: 7; 2025: 3; 2026: 2.

36 papers.

Sort:

By area:

  1. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  2. 2023BiFormer: Vision Transformer with Bi-Level Routing Attentioncited by 150
  3. 2023Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Groupingcited by 92
  4. 2023RMT: Retentive Networks Meet Vision Transformerscited by 68
  5. 2022Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarioscited by 65
  6. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 47
  7. 2023Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoisingcited by 35
  8. 2024Breaking the Low-Rank Dilemma of Linear Attentioncited by 31
  9. 2023Recursive Generalization Transformer for Image Super-Resolutioncited by 28
  10. 2023Lightweight Vision Transformer with Bidirectional Interactioncited by 25
  11. 2023DAT++: Spatially Dynamic Vision Transformer with Deformable Attentioncited by 21
  12. 2023TransXNet: Learning Both Global and Local Dynamics with a Dual Dynamic Token Mixer for Visual Recognitioncited by 21
24 more
  1. 2023CrossFormer++: A Versatile Vision Transformer Hinging on Cross-scale Attentioncited by 18
  2. 2024Embedding-Free Transformer with Inference Spatial Reduction for Efficient Semantic Segmentationcited by 14
  3. 2023BoxSnake: Polygonal Instance Segmentation with Box Supervisioncited by 12
  4. 2024HIRI-ViT: Scaling Vision Transformer with High Resolution Inputscited by 12
  5. 2025HaploVL: A Single-Transformer Baseline for Multi-Modal Understandingcited by 11
  6. 2023UniHead: Unifying Multi-Perception for Detection Headscited by 9
  7. 2024Vision Transformer with Sparse Scan Priorcited by 6
  8. 2023SSGD: A smartphone screen glass dataset for defect detectioncited by 5
  9. 2024Semantic Equitable Clustering: A Simple and Effective Strategy for Clustering Vision Tokenscited by 5
  10. 2024DeBiFormer: Vision Transformer with Deformable Agent Bi-level Routing Attentioncited by 4
  11. 2023Vision Backbone Enhancement via Multi-Stage Cross-Scale Attentioncited by 3
  12. 2024FViT: A Focal Vision Transformer with Gabor Filtercited by 3
  13. 2022UniInst: Unique Representation for End-to-End Instance Segmentationcited by 2
  14. 2023Vision Transformer with Attention Map Hallucination and FFN Compactioncited by 2
  15. 2023Cross-level Attention with Overlapped Windows for Camouflaged Object Detectioncited by 2
  16. 2026SAT: Selective Aggregation Transformer for Image Super-Resolutioncited by 2
  17. 2022Masked autoencoders are effective solution to transformer data-hungrycited by 1
  18. 2023EViT: An Eagle Vision Transformer with Bi-Fovea Self-Attentioncited by 1
  19. 2023Bitformer: An efficient Transformer with bitwise operation-based attention for Big Data Analytics at low-cost low-precision devicescited by 1
  20. 2025Learning Spatial Decay for Vision Transformerscited by 1
  21. 2023APPT : Asymmetric Parallel Point Transformer for 3D Point Cloud Understandingcited by 0
  22. 2023An End-to-End Network for Upright Adjustment of Panoramic Imagescited by 0
  23. 2025FusDreamer: Label-efficient Remote Sensing World Model for Multimodal Data Classificationcited by 0
  24. 2026Advancing Vision Transformer with Enhanced Spatial Priorscited by 0