mlsanityData to 7 Oct 2026

Unified Visual Transformer Compression

arXiv 2203.08243 ↗First posted 15 Mar 2022cs.LG cs.CVcited by 79

Not on the map. Why?

Abstract (arXiv)

Vision transformers (ViTs) have gained popularity recently. Even without customized image operators such as convolutions, ViTs can yield competitive performance when properly trained on massive data. However, the computational overhead of ViTs remains prohibitive, due to stacking multi-head self-attention modules and else. Compared to the vast literature and prevailing success in compressing convolutional neural networks, the study of Vision Transformer compression has also just emerged, and existing works focused on one or two aspects of compression. This paper proposes a unified ViT compression framework that seamlessly assembles three effective techniques: pruning, layer skipping, and knowledge distillation. We formulate a budget-constrained, end-to-end optimization framework, targeting jointly learning model weights, layer-wise pruning ratios/masks, and skip configurations, under a distillation loss. The optimization problem is then solved using the primal-dual algorithm. Experiments are conducted with several ViT variants, e.g. DeiT and T2T-ViT backbones on the ImageNet dataset, and our approach consistently outperforms recent competitors. For example, DeiT-Tiny can be trimmed down to 50% of the original FLOPs almost without losing accuracy. Codes are available online:~urlhttps://github.com/VITA-Group/UVC.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Builds on 64 of its 81 references that are in the map. First posted on arXiv 15 Mar 2022.

Builds on

64 of 81 references.

References · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
38 more
  1. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  2. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  3. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  4. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  6. 2020Point Transformercited by 1,462
  7. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  8. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  9. 2020Pre-Trained Image Processing Transformercited by 891
  10. 2019Are Sixteen Heads Really Better than One?cited by 803
  11. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  12. 2018Image Transformercited by 770
  13. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  14. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  15. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  16. 2020A Survey on Visual Transformercited by 653
  17. 2019The State of Sparsity in Deep Neural Networkscited by 471
  18. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  19. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  20. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  21. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  22. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 375
  23. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
  24. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  25. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 346
  26. 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 291
  27. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  28. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  29. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  30. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
  31. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  32. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  33. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  34. 2021TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Upcited by 162
  35. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  36. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  37. 2021Patch Slimming for Efficient Vision Transformerscited by 131
  38. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 19; 2024: 32; 2025: 13; 2026: 5.

79 papers.

Sort:

By area:

  1. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  2. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  3. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  4. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  5. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
  6. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 66
  7. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformerscited by 46
  8. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
  9. 2024A Survey on Transformer Compressioncited by 44
  10. 2022Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attentioncited by 35
  11. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  12. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
38 more
  1. 2022ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attentioncited by 29
  2. 2022PPT: token-Pruned Pose Transformer for monocular and multi-view human pose estimationcited by 27
  3. 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26
  4. 2022Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Trainingcited by 24
  5. 2025Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategiescited by 24
  6. 2024Isomorphic Pruning for Vision Modelscited by 21
  7. 2023Boost Vision Transformer with GPU-Friendly Sparsity and Quantizationcited by 20
  8. 2024OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decompositioncited by 19
  9. 2024PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptationcited by 17
  10. 2022Attention Distillation: self-supervised vision transformer students need more guidancecited by 16
  11. 2024The Need for Speed: Pruning Transformers with One Recipecited by 15
  12. 2024LPViT: Low-Power Semi-structured Pruning for Vision Transformerscited by 15
  13. 2024Progressive distillation induces an implicit curriculumcited by 15
  14. 2023COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Modelscited by 14
  15. 2024Comprehensive Survey of Model Compression and Speed up for Vision Transformerscited by 14
  16. 2023Knowledge Distillation in Vision Transformers: A Critical Reviewcited by 13
  17. 2023GOHSP: A Unified Framework of Graph and Optimization-based Heterogeneous Structured Pruning for Vision Transformercited by 10
  18. 2023CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTscited by 9
  19. 2023The Role of Masking for Efficient Supervised Knowledge Distillation of Vision Transformerscited by 8
  20. 2024Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compressioncited by 8
  21. 2024Dense Vision Transformer Compression with Few Samplescited by 8
  22. 2022Data Level Lottery Ticket Hypothesis for Vision Transformerscited by 7
  23. 2024Efficient Visual Transformer by Learnable Token Mergingcited by 7
  24. 2024Data-independent Module-aware Pruning for Hierarchical Vision Transformerscited by 6
  25. 2025Longitudinal Flow Matching for Trajectory Modelingcited by 6
  26. 2024Accelerating ViT Inference on FPGA through Static and Dynamic Pruningcited by 5
  27. 2024SNP: Structured Neuron-level Pruning to Preserve Attention Scorescited by 5
  28. 2024Quantized Prompt for Efficient Generalization of Vision-Language Modelscited by 5
  29. 2025RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layerscited by 5
  30. 2023RingMo-lite: A Remote Sensing Multi-task Lightweight Network with CNN-Transformer Hybrid Frameworkcited by 4
  31. 2024Block Selective Reprogramming for On-device Training of Vision Transformerscited by 4
  32. 2024Compress and Compare: Interactively Evaluating Efficiency and Behavior Across ML Model Compression Experimentscited by 4
  33. 2025MDP: Multidimensional Vision Model Pruning with Latency Constraintcited by 4
  34. 2023Resource Constrained Model Compression via Minimax Optimization for Spiking Neural Networkscited by 3
  35. 2023FLORA: Fine-grained Low-Rank Architecture Search for Vision Transformercited by 3
  36. 2024Automatic Channel Pruning for Multi-Head Attentioncited by 3
  37. 2024Mew: Multiplexed Immunofluorescence Image Analysis through an Efficient Multiplex Networkcited by 3
  38. 2025Diversity-Guided MLP Reduction for Efficient Large Vision Transformerscited by 3

50 most cited of 79 All 79 (JSON)