mlsanityData to 14 Oct 2026

Lite Vision Transformer with Enhanced Self-Attention

arXiv 2112.10809 ↗First posted 20 Dec 2021cs.CVcited by 64

Not on the map. Why?

Abstract (arXiv)

Despite the impressive representation capacity of vision transformer models, current light-weight vision transformer models still suffer from inconsistent and incorrect dense predictions at local regions. We suspect that the power of their self-attention mechanism is limited in shallower and thinner networks. We propose Lite Vision Transformer (LVT), a novel light-weight transformer network with two enhanced self-attention mechanisms to improve the model performances for mobile deployment. For the low-level features, we introduce Convolutional Self-Attention (CSA). Unlike previous approaches of merging convolution and self-attention, CSA introduces local self-attention into the convolution within a kernel of size 3x3 to enrich low-level features in the first stage of LVT. For the high-level features, we propose Recursive Atrous Self-Attention (RASA), which utilizes the multi-scale context when calculating the similarity map and a recursive mechanism to increase the representation capability with marginal extra parameter cost. The superiority of LVT is demonstrated on ImageNet recognition, ADE20K semantic segmentation, and COCO panoptic segmentation. The code is made publicly available.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Builds on 59 of its 70 references that are in the map. First posted on arXiv 20 Dec 2021.

Builds on

59 of 70 references.

References · 59

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  8. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  10. 2017Squeeze-and-Excitation Networkscited by 4,789
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
38 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  2. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  3. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  4. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  5. 2016Pyramid Scene Parsing Networkcited by 2,479
  6. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  7. 2017Non-local Neural Networkscited by 2,293
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  9. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  11. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  12. 2017Random Erasing Data Augmentationcited by 1,239
  13. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  14. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  15. 2016Deep Networks with Stochastic Depthcited by 979
  16. 2020Designing Network Design Spacescited by 960
  17. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  18. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  19. 2021Going deeper with Image Transformerscited by 764
  20. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  21. 2019Panoptic Feature Pyramid Networkscited by 645
  22. 2021Transformer in Transformercited by 633
  23. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  24. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  25. 2014Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFscited by 554
  26. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  27. 2016RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentationcited by 465
  28. 2021Conditional Positional Encodings for Vision Transformerscited by 406
  29. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  30. 2019Attention Augmented Convolutional Networkscited by 363
  31. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  32. 2015Deeply-Recursive Convolutional Network for Image Super-Resolutioncited by 317
  33. 2020Exploring Self-attention for Image Recognitioncited by 313
  34. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  35. 2021Mobile-Former: Bridging MobileNet and Transformercited by 235
  36. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  37. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  38. 2020DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolutioncited by 222

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 17; 2023: 18; 2024: 15; 2025: 8; 2026: 6.

64 papers.

Sort:

By area:

  1. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  2. 2024EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mambacited by 146
  3. 2023RMT: Retentive Networks Meet Vision Transformerscited by 68
  4. 2023Omni Aggregation Networks for Lightweight Image Super-Resolutioncited by 62
  5. 2022Hydra Attention: Efficient Attention with Many Headscited by 55
  6. 2022MOAT: Alternating Mobile Convolution and Attention Brings Strong Vision Modelscited by 47
  7. 2022MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Featurescited by 44
  8. 2022LightViT: Towards Light-Weight Convolution-Free Vision Transformerscited by 40
  9. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  10. 2023Dynamic Perceiver for Efficient Visual Recognitioncited by 33
  11. 2023TransFlow: Transformer as Flow Learnercited by 32
  12. 2022A Closer Look at Self-Supervised Lightweight Vision Transformerscited by 31
38 more
  1. 2024SegMAN: Omni-scale Context Modeling with State Space Models and Local Attention for Semantic Segmentationcited by 31
  2. 2022EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithmcited by 27
  3. 2022Learning Target-aware Representation for Visual Tracking via Informative Interactionscited by 25
  4. 2023Lightweight Vision Transformer with Bidirectional Interactioncited by 25
  5. 2024Are Self-Attentions Effective for Time Series Forecasting?cited by 19
  6. 2023SeaFormer++: Squeeze-enhanced Axial Transformer for Mobile Visual Recognitioncited by 18
  7. 2022TransKD: Transformer Knowledge Distillation for Efficient Semantic Segmentationcited by 17
  8. 2024A Comprehensive Survey of Convolutions in Deep Learning: Applications, Challenges, and Future Trendscited by 16
  9. 2024Embedding-Free Transformer with Inference Spatial Reduction for Efficient Semantic Segmentationcited by 14
  10. 2022kMaX-DeepLab: k-means Mask Transformercited by 12
  11. 2024MetaSeg: MetaFormer-based Global Contexts-aware Network for Efficient Semantic Segmentationcited by 12
  12. 2022BViT: Broad Attention based Vision Transformercited by 10
  13. 2023Isomer: Isomerous Transformer for Zero-shot Video Object Segmentationcited by 10
  14. 2022Self-Attentive Pooling for Efficient Deep Learningcited by 9
  15. 2023U-MixFormer: UNet-like Transformer with Mix-Attention for Efficient Semantic Segmentationcited by 8
  16. 2024Vim-F: Visual State Space Model Benefiting from Learning in the Frequency Domaincited by 8
  17. 2024Data-independent Module-aware Pruning for Hierarchical Vision Transformerscited by 6
  18. 2024Vision Transformer with Sparse Scan Priorcited by 6
  19. 2024Efficient Partitioning Vision Transformer on Edge Devices for Distributed Inferencecited by 6
  20. 2022Lightweight Vision Transformer with Cross Feature Attentioncited by 5
  21. 2023SwinCross: Cross-modal Swin Transformer for Head-and-Neck Tumor Segmentation in PET/CT Imagescited by 5
  22. 2023Rethinking Local Perception in Lightweight Vision Transformercited by 5
  23. 2023PSLT: A Light-weight Vision Transformer with Ladder Self-Attention and Progressive Shiftcited by 5
  24. 2024Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologiescited by 5
  25. 2022Deeply Supervised Skin Lesions Diagnosis with Stage and Branch Attentioncited by 4
  26. 2024Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insightscited by 4
  27. 2025Janus: Collaborative Vision Transformer Under Dynamic Network Environmentcited by 4
  28. 2025Improving underwater semantic segmentation with underwater image quality attention and muti-scale aggregation attentioncited by 4
  29. 2025PromptSR: Cascade Prompting for Lightweight Image Super-Resolutioncited by 4
  30. 2023Vision Backbone Enhancement via Multi-Stage Cross-Scale Attentioncited by 3
  31. 2024MacFormer: Semantic Segmentation with Fine Object Boundariescited by 3
  32. 2026ATD: Improved Transformer with Adaptive Token Dictionary for Image Restorationcited by 3
  33. 2023Enhancing Representations through Heterogeneous Self-Supervised Learningcited by 2
  34. 2026STENet: Superpixel Token Enhancing Network for RGB-D Salient Object Detectioncited by 2
  35. 2023DualToken-ViT: Position-aware Efficient Vision Transformer with Dual Token Fusioncited by 1
  36. 2025Distilling Textual Priors from LLM to Efficient Image Fusioncited by 1
  37. 2026GeoCueFormer: Geometry-Guided Wavelet Representation and Prediction-Cued Dual-Stage Decoder for Underwater Semantic Segmentationcited by 1
  38. 2022MaiT: Leverage Attention Masks for More Efficient Image Transformerscited by 0

50 most cited of 64 All 64 (JSON)