mlsanityData to 14 Oct 2026

Comprehensive Survey of Model Compression and Speed up for Vision Transformers

arXiv 2404.10407 ↗First posted 16 Apr 2024cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (ViT) have marked a paradigm shift in computer vision, outperforming state-of-the-art models across diverse tasks. However, their practical deployment is hampered by high computational and memory demands. This study addresses the challenge by evaluating four primary model compression techniques: quantization, low-rank approximation, knowledge distillation, and pruning. We methodically analyze and compare the efficacy of these techniques and their combinations in optimizing ViTs for resource-constrained environments. Our comprehensive experimental evaluation demonstrates that these methods facilitate a balanced compromise between model accuracy and computational efficiency, paving the way for wider application in edge computing devices.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 31 of its 52 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

31 of 52 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  5. 2020Rethinking Attention with Performerscited by 1,601
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  8. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  9. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  10. 2019Learned Step Size Quantizationcited by 628
  11. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  12. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
19 more
  1. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  2. 2021Post-Training Quantization for Vision Transformercited by 278
  3. 2021Patch Slimming for Efficient Vision Transformerscited by 133
  4. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 130
  5. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  6. 2021Vision Transformer Pruningcited by 79
  7. 2022Unified Visual Transformer Compressioncited by 79
  8. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 45
  9. 2021A Unified Pruning Framework for Vision Transformerscited by 40
  10. 2020Circumventing Outliers of AutoAugment with Knowledge Distillationcited by 29
  11. 2024DDN-SLAM: Real-time Dense Dynamic Neural Implicit SLAMcited by 23
  12. 2021BridgeDPI: A Novel Graph Neural Network for Predicting Drug-Protein Interactionscited by 12
  13. 2018Assessing four Neural Networks on Handwritten Digit Recognition Dataset (MNIST)cited by 8
  14. 2024Jointly Learning Selection Matrices For Transmitters, Receivers And Fourier Coefficients In Multichannel Imagingcited by 7
  15. 2019Deep Learning-Based Strategy for Macromolecules Classification with Imbalanced Data from Cellular Electron Cryotomographycited by 4
  16. 2018Learning Robust Heterogeneous Signal Features from Parallel Neural Network for Audio Sentiment Analysiscited by 3
  17. 2024Machine learning-based system reliability analysis with Gaussian Process Regressioncited by 2
  18. 2019The Application of Bipartite Matching in Assignment Problemcited by 1
  19. 2019An efficient sorting algorithm - Ultimate Heapsort(UHS)cited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 3.

14 papers.

Sort:

By area:

  1. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  2. 2025AutoNeural: Co-Designing Vision-Language Models for NPU Inferencecited by 17
  3. 2024VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentationcited by 11
  4. 2024Cross-Modal Domain Adaptation in Brain Disease Diagnosis: Maximum Mean Discrepancy-based Convolutional Neural Networkscited by 2
  5. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  6. 2024A Comparative Study of Pruning Methods in Transformer-based Time Series Forecastingcited by 1
  7. 2025Back to Fundamentals: Low-Level Visual Features Guided Progressive Token Pruningcited by 1
  8. 2024CFPFormer: Feature-pyramid like Transformer Decoder for Segmentation and Detectioncited by 0
  9. 2024Survey: Transformer-based Models in Data Modality Conversioncited by 0
  10. 2025TuneComp: Joint Fine-tuning and Compression for Large Foundation Modelscited by 0
  11. 2025Optimizing Resource Allocation for Geographically-Distributed Inference by Large Language Modelscited by 0
  12. 2026Recti-Q: Feature-Space Rectification for Out-of-Distribution-Robust Quantized Perception in Edge Roboticscited by 0
2 more
  1. 2026Are the High-weight Neurons the Important Ones in Image Classification Neural Networks?cited by 0
  2. 2026MixFrag: Fragility-Guided Mixed-Precision Post-Training Quantization for Vision Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Compressing Vision Transformers for Low-Resource Visual Learning
  2. 2025Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AI
  3. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking
  4. 2024Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey
  5. 2023CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs
  6. 2026ButterflyViT: 354× Expert Compression for Edge Vision Transformers
  7. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer
  8. 2024ViT-1.58b: Mobile Vision Transformers in the 1-bit Era