mlsanityData to 7 Oct 2026

LocalViT: Analyzing Locality in Vision Transformers

arXiv 2104.05707 ↗First posted 12 Apr 2021cs.CVcited by 228

Not on the map. Why?

Abstract (arXiv)

The aim of this paper is to study the influence of locality mechanisms in vision transformers. Transformers originated from machine translation and are particularly good at modelling long-range dependencies within a long sequence. Although the global interaction between the token embeddings could be well modelled by the self-attention mechanism of transformers, what is lacking is a locality mechanism for information exchange within a local region. In this paper, locality mechanism is systematically investigated by carefully designed controlled experiments. We add locality to vision transformers into the feed-forward network. This seemingly simple solution is inspired by the comparison between feed-forward networks and inverted residual blocks. The importance of locality mechanisms is validated in two ways: 1) A wide range of design choices (activation function, layer placement, expansion ratio) are available for incorporating locality mechanisms and proper choices can lead to a performance gain over the baseline, and 2) The same locality mechanism is successfully applied to vision transformers with different architecture designs, which shows the generalization of the locality concept. For ImageNet2012 classification, the locality-enhanced transformers outperform the baselines Swin-T, DeiT-T, and PVT-T by 1.0%, 2.6% and 3.1% with a negligible increase in the number of parameters and computational effort. Code is available at https://github.com/ofsoundof/LocalViT.

Key facts. Cited by 228 papers in the map (data through 7 Oct 2026). Builds on 30 of its 45 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

30 of 45 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  10. 2017Squeeze-and-Excitation Networkscited by 4,763
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
18 more
  1. 2017Non-local Neural Networkscited by 2,290
  2. 2019Searching for MobileNetV3cited by 1,984
  3. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  4. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  5. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  6. 2020Designing Network Design Spacescited by 954
  7. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  8. 2021Transformer in Transformercited by 632
  9. 2019ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networkscited by 572
  10. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  11. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  12. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  13. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  14. 2018Non-Local Recurrent Network for Image Restorationcited by 160
  15. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  16. 2020Lite Transformer with Long-Short Range Attentioncited by 127
  17. 2021BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Searchcited by 63
  18. 2020DeLighT: Deep and Light-weight Transformercited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 56; 2022: 73; 2023: 55; 2024: 30; 2025: 9; 2026: 5.

228 papers.

Sort:

By area:

  1. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  2. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,869
  3. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  4. 2021Uformer: A General U-Shaped Transformer for Image Restorationcited by 864
  5. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  6. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  7. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  8. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  9. 2022Activating More Pixels in Image Super-Resolution Transformercited by 433
  10. 2022MixFormer: End-to-End Tracking with Iterative Mixed Attentioncited by 278
  11. 2022Vision Transformers for Single Image Dehazingcited by 255
  12. 2021TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditionscited by 243
38 more
  1. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  2. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  3. 2022VRT: A Video Restoration Transformercited by 219
  4. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  5. 2023Learning A Sparse Transformer Network for Effective Image Derainingcited by 204
  6. 2022Medical Image Segmentation Review: The success of U-Netcited by 190
  7. 2022Focal Modulation Networkscited by 186
  8. 2022DaViT: Dual Attention Vision Transformerscited by 174
  9. 2022Recurrent Video Restoration Transformer with Guided Deformable Attentioncited by 168
  10. 2022Separable Self-attention for Mobile Vision Transformerscited by 163
  11. 2022Efficient Long-Range Attention Network for Image Super-resolutioncited by 151
  12. 2021Transformer for Single Image Super-Resolutioncited by 150
  13. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  14. 2023Efficient and Explicit Modelling of Image Hierarchies for Image Restorationcited by 145
  15. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
  16. 2021Towards Robust Vision Transformercited by 137
  17. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 126
  18. 2021Vision Transformer for Small-Size Datasetscited by 122
  19. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  20. 2021Video Super-Resolution Transformercited by 120
  21. 2021A Survey of Visual Transformerscited by 117
  22. 2022ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyondcited by 117
  23. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  24. 2021Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformercited by 115
  25. 2021Efficient Training of Visual Transformers with Small Datasetscited by 107
  26. 2021MISSFormer: An Effective Medical Image Segmentation Transformercited by 103
  27. 2021AS-MLP: An Axial Shifted MLP Architecture for Visioncited by 101
  28. 2021Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understandingcited by 97
  29. 2023Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolationcited by 97
  30. 2022Filter-enhanced MLP is All You Need for Sequential Recommendationcited by 96
  31. 2022PanoFormer: Panorama Transformer for Indoor 360 Depth Estimationcited by 94
  32. 2021Multi-Scale High-Resolution Vision Transformer for Semantic Segmentationcited by 92
  33. 2022Practical Blind Image Denoising via Swin-Conv-UNet and Data Synthesiscited by 85
  34. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 79
  35. 2023GET: Group Event Transformer for Event-Based Visioncited by 73
  36. 2021KVT: k-NN Attention for Boosting Vision Transformerscited by 64
  37. 2021Lite Vision Transformer with Enhanced Self-Attentioncited by 64
  38. 2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63

50 most cited of 228 All 228 (JSON)