mlsanityData to 14 Oct 2026

Structured Pruning Adapters

arXiv 2211.10155 ↗First posted 17 Nov 2022cs.CV cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Adapters are a parameter-efficient alternative to fine-tuning, which augment a frozen base network to learn new tasks. Yet, the inference of the adapted model is often slower than the corresponding fine-tuned model. To improve on this, we propose Structured Pruning Adapters (SPAs), a family of compressing, task-switching network adapters, that accelerate and specialize networks using tiny parameter sets and structured pruning. Specifically, we propose a channel-based SPA and evaluate it with a suite of pruning methods on multiple computer vision benchmarks. Compared to regular structured pruning with fine-tuning, our channel-SPAs improve accuracy by 6.9% on average while using half the parameters at 90% pruned weights. Alternatively, they can learn adaptations with 17x fewer parameters at 70% pruning with 1.6% lower accuracy. Similarly, our block-SPA requires far fewer parameters than pruning with fine-tuning. Our experimental code and Python library of adapters are available at github.com/lukashedegaard/structured-pruning-adapters.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 41 of its 57 references that are in the map. First posted on arXiv 17 Nov 2022.

Builds on

41 of 57 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
29 more
  1. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  3. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  4. 2020Knowledge Distillation: A Surveycited by 1,603
  5. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  6. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  7. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  8. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  9. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 637
  10. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  11. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  12. 2017Learning multiple visual domains with residual adapterscited by 557
  13. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  14. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  15. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  16. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
  17. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 237
  18. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  19. 2019Structured Pruning of Large Language Modelscited by 203
  20. 2021Block Pruning For Faster Transformerscited by 177
  21. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  22. 2019Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruningcited by 89
  23. 2021Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameterscited by 66
  24. 2021Counter-Interference Adapter for Multilingual Machine Translationcited by 40
  25. 2017meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfittingcited by 34
  26. 2021SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruningcited by 30
  27. 2022Continual Transformers: Redundancy-Free Attention for Online Inferencecited by 12
  28. 2021Continual 3D Convolutional Neural Networks for Real-time Processing of Videoscited by 9
  29. 2017Improving Efficiency in Convolutional Neural Network with Multilinear Filterscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 4; 2026: 2.

12 papers.

  1. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  3. 2023Personalize Segment Anything Model with One Shotcited by 250
  4. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 23
  5. 2025LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuningcited by 4
  6. 2023Parameter-Efficient Fine-Tuning with Layer Pruning on Free-Text Sequence-to-Sequence Modelingcited by 3
  7. 2025Long Exposure: Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsitycited by 2
  8. 2025LOBERT: Generative AI Foundation Model for Limit Order Book Messagescited by 2
  9. 2025CURing Large Models: Compression via CUR Decompositioncited by 1
  10. 2026An Adapter-free Fine-tuning Approach for Tuning 3D Foundation Modelscited by 1
  11. 2023Efficient Online Processing with Deep Neural Networkscited by 0
  12. 2026Inheritance Between Feedforward and Convolutional Networks via Model Projectioncited by 0