mlsanityData to 7 Oct 2026

Automatic Channel Pruning for Multi-Head Attention

arXiv 2405.20867 ↗First posted 31 May 2024cs.CV cs.AI cs.CCcited by 3

Not on the map. Why?

Abstract (arXiv)

Despite the strong performance of Transformers, their quadratic computation complexity presents challenges in applying them to vision tasks. Automatic pruning is one of effective methods for reducing computation complexity without heuristic approaches. However, directly applying it to multi-head attention is not straightforward due to channel misalignment. In this paper, we propose an automatic channel pruning method to take into account the multi-head attention mechanism. First, we incorporate channel similarity-based weights into the pruning indicator to preserve more informative channels in each head. Then, we adjust pruning indicator to enforce removal of channels in equal proportions across all heads, preventing the channel misalignment. We also add a reweight module to compensate for information loss resulting from channel removal, and an effective initialization step for pruning indicator based on difference of attention between original structure and each channel. Our proposed method can be used to not only original attention, but also linear attention, which is more efficient as linear complexity with respect to the number of tokens. On ImageNet-1K, applying our pruning method to the FLattenTransformer, which includes both attention mechanisms, shows outperformed accuracy for several MACs compared with previous state-of-the-art efficient models and pruned methods. Code will be available soon.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 40 of its 45 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

40 of 45 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  7. 2020Reformer: The Efficient Transformercited by 1,658
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  10. 2022Token Merging: Your ViT But Fastercited by 995
  11. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  12. 2021Going deeper with Image Transformerscited by 759
28 more
  1. 2021Transformer in Transformercited by 632
  2. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  3. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  4. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  5. 2022Vision Transformer with Deformable Attentioncited by 293
  6. 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 291
  7. 2022Neighborhood Attention Transformercited by 235
  8. 2019MetaPruning: Meta Learning for Automatic Neural Network Channel Pruningcited by 226
  9. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  10. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  11. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
  12. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  13. 2022EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applicationscited by 121
  14. 2022Unified Visual Transformer Compressioncited by 79
  15. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 79
  16. 2021Vision Transformer Pruningcited by 77
  17. 2021Global Vision Transformer Pruning with Hessian-Aware Saliencycited by 69
  18. 2023SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applicationscited by 66
  19. 2023Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentationcited by 61
  20. 2020DHP: Differentiable Meta Pruning via HyperNetworkscited by 57
  21. 2021GLiT: Neural Architecture Search for Global and Local Image Transformercited by 53
  22. 2021Differentiable Subset Pruning of Transformer Headscited by 46
  23. 2022CHEX: CHannel EXploration for CNN Model Compressioncited by 36
  24. 2022Pruning-as-Search: Efficient Neural Architecture Search via Channel Pruning and Structural Reparameterizationcited by 31
  25. 2022Castling-ViT: Compressing Self-Attention via Switching Towards Linear-Angular Attention at Vision Transformer Inferencecited by 28
  26. 2021Pruning Self-attentions into Convolutional Layers in Single Pathcited by 24
  27. 2022Auto-scaling Vision Transformers without Trainingcited by 15
  28. 2023A Unified Framework for Soft Threshold Pruningcited by 12

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024Supervised Fine-Tuning Achieve Rapid Task Adaption Via Alternating Attention Head Activation Patternscited by 5
  2. 2025HEART-VIT: Hessian-Guided Efficient Dynamic Attention and Token Pruning in Vision Transformercited by 2
  3. 2025Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspectivecited by 0