mlsanityData to 7 Oct 2026

Image Recognition with Online Lightweight Vision Transformer: A Survey

arXiv 2505.03113 ↗First posted 6 May 2025cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

The Transformer architecture has achieved significant success in natural language processing, motivating its adaptation to computer vision tasks. Unlike convolutional neural networks, vision transformers inherently capture long-range dependencies and enable parallel processing, yet lack inductive biases and efficiency benefits, facing significant computational and memory challenges that limit its real-world applicability. This paper surveys various online strategies for generating lightweight vision transformers for image recognition, focusing on three key areas: Efficient Component Design, Dynamic Network, and Knowledge Distillation. We evaluate the relevant exploration for each topic on the ImageNet-1K benchmark, analyzing trade-offs among precision, parameters, throughput, and more to highlight their respective advantages, disadvantages, and flexibility. Finally, we propose future research directions and potential challenges in the lightweighting of vision transformers with the aim of inspiring further exploration and providing practical guidance for the community. Project Page: https://github.com/ajxklo/Lightweight-VIT

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 142 of its 213 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

142 of 213 references.

References · 142

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  6. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  7. 2020Reformer: The Efficient Transformercited by 1,658
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2020Knowledge Distillation: A Surveycited by 1,578
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  12. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
38 more
  1. 2021Transformers in Vision: A Surveycited by 1,009
  2. 2022Token Merging: Your ViT But Fastercited by 995
  3. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  4. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  5. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  6. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  7. 2021Transformer in Transformercited by 632
  8. 2019Deep Equilibrium Modelscited by 603
  9. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  10. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  11. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  12. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  13. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  14. 2021Pay Attention to MLPscited by 367
  15. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  16. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  17. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  18. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  19. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  20. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  21. 2022Long Range Language Modeling via Gated State Spacescited by 265
  22. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 260
  23. 2021Global Filter Networks for Image Classificationcited by 257
  24. 2021Mobile-Former: Bridging MobileNet and Transformercited by 234
  25. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  26. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  27. 2019Depth-Adaptive Transformercited by 227
  28. 2024Rotary Position Embedding for Vision Transformercited by 226
  29. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  30. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  31. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  32. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  33. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
  34. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  35. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 164
  36. 2022Separable Self-attention for Mobile Vision Transformerscited by 163
  37. 2023BiFormer: Vision Transformer with Bi-Level Routing Attentioncited by 150
  38. 2024EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mambacited by 146

50 most cited of 142 All 142 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectorscited by 1
  2. 2025Exploring Kolmogorov-Arnold Network Expansions in Vision Transformers for Mitigating Catastrophic Forgetting in Continual Learningcited by 0