mlsanityData to 14 Oct 2026

Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space

arXiv 2201.00814 ↗First posted 3 Jan 2022cs.CV cs.AI cs.LGcited by 66

Not on the map. Why?

Abstract (arXiv)

This paper explores the feasibility of finding an optimal sub-model from a vision transformer and introduces a pure vision transformer slimming (ViT-Slim) framework. It can search a sub-structure from the original model end-to-end across multiple dimensions, including the input tokens, MHSA and MLP modules with state-of-the-art performance. Our method is based on a learnable and unified ℓ1 sparsity constraint with pre-defined factors to reflect the global importance in the continuous searching space of different dimensions. The searching process is highly efficient through a single-shot training scheme. For instance, on DeiT-S, ViT-Slim only takes ~43 GPU hours for the searching process, and the searched structure is flexible with diverse dimensionalities in different modules. Then, a budget threshold is employed according to the requirements of accuracy-FLOPs trade-off on running devices, and a re-training process is performed to obtain the final model. The extensive experiments show that our ViT-Slim can compress up to 40% of parameters and 40% FLOPs on various vision transformers while increasing the accuracy by ~0.6% on ImageNet. We also demonstrate the advantage of our searched models on several downstream datasets. Our code is available at https://github.com/Arnav0400/ViT-Slim.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Builds on 61 of its 67 references that are in the map. First posted on arXiv 3 Jan 2022.

Builds on

61 of 67 references.

References · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  8. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  10. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  11. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  12. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
38 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  2. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  3. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  5. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  6. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  7. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  8. 2017Random Erasing Data Augmentationcited by 1,239
  9. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  10. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  11. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  12. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  13. 2019When Does Label Smoothing Help?cited by 988
  14. 2016Deep Networks with Stochastic Depthcited by 979
  15. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  16. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  17. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  18. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  19. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  20. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  21. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  22. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 485
  23. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  24. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  25. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  26. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 414
  27. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 378
  28. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 334
  29. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 188
  30. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 188
  31. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 184
  32. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 183
  33. 2020Rethinking Transformer-based Set Prediction for Object Detectioncited by 156
  34. 2018NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applicationscited by 147
  35. 2021Patch Slimming for Efficient Vision Transformerscited by 133
  36. 2019FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Searchcited by 130
  37. 2018Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layerscited by 112
  38. 2019XNOR-Net++: Improved Binary Neural Networkscited by 97

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 17; 2024: 23; 2025: 7; 2026: 10.

66 papers.

Sort:

By area:

  1. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 241
  2. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
  3. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  4. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  5. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  6. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  7. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 85
  8. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 77
  9. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 68
  10. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformerscited by 60
  11. 2023MixFormerV2: Efficient Fully Transformer Trackingcited by 57
  12. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 54
38 more
  1. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 54
  2. 2024A Survey on Transformer Compressioncited by 46
  3. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  4. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  5. 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33
  6. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  7. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 21
  8. 2024Isomorphic Pruning for Vision Modelscited by 21
  9. 2023Boost Vision Transformer with GPU-Friendly Sparsity and Quantizationcited by 20
  10. 2024OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decompositioncited by 20
  11. 2024PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptationcited by 17
  12. 2022Super Vision Transformercited by 13
  13. 2024MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruningcited by 13
  14. 2024Agglomerative Token Clusteringcited by 13
  15. 2022Greybox XAI: a Neural-Symbolic learning framework to produce interpretable predictions for image classificationcited by 12
  16. 2024HydraViT: Stacking Heads for a Scalable ViTcited by 11
  17. 2024PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inferencecited by 10
  18. 2024General Compression Framework for Efficient Transformer Object Trackingcited by 10
  19. 2024Dense Vision Transformer Compression with Few Samplescited by 9
  20. 2022Spartan: Differentiable Sparsity via Regularized Transportationcited by 8
  21. 2024Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compressioncited by 8
  22. 2022Data Level Lottery Ticket Hypothesis for Vision Transformerscited by 7
  23. 2024Efficient Visual Transformer by Learnable Token Mergingcited by 7
  24. 2025Discovering Influential Neuron Path in Vision Transformerscited by 7
  25. 2024Data-independent Module-aware Pruning for Hierarchical Vision Transformerscited by 6
  26. 2024Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuningcited by 6
  27. 2023Three Guidelines You Should Know for Universally Slimmable Self-Supervised Learningcited by 5
  28. 2022Slimmable Networks for Contrastive Self-supervised Learningcited by 4
  29. 2024Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformercited by 4
  30. 2024Neural Metamorphosiscited by 4
  31. 2025One-Shot Knowledge Transfer for Scalable Person Re-Identificationcited by 4
  32. 2023Efficient Transformer-based 3D Object Detection with Dynamic Token Haltingcited by 3
  33. 2025Diversity-Guided MLP Reduction for Efficient Large Vision Transformerscited by 3
  34. 2023Spatiotemporal Attention-based Semantic Compression for Real-time Video Recognitioncited by 2
  35. 2024Surgical Feature-Space Decomposition of LLMs: Why, When and How?cited by 2
  36. 2024Beyond 2:4: exploring V:N:M sparsity for efficient transformer inference on GPUscited by 2
  37. 2025Adaptive Pruning of Pretrained Transformer via Differential Inclusionscited by 2
  38. 2026Adaptive MLP Pruning for Large Vision Transformerscited by 2

50 most cited of 66 All 66 (JSON)