mlsanityData to 7 Oct 2026

Adaptive Pruning of Pretrained Transformer via Differential Inclusions

arXiv 2501.03289 ↗First posted 6 Jan 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Large transformers have demonstrated remarkable success, making it necessary to compress these models to reduce inference costs while preserving their perfor-mance. Current compression algorithms prune transformers at fixed compression ratios, requiring a unique pruning process for each ratio, which results in high computational costs. In contrast, we propose pruning of pretrained transformers at any desired ratio within a single pruning stage, based on a differential inclusion for a mask parameter. This dynamic can generate the whole regularization solution path of the mask parameter, whose support set identifies the network structure. Therefore, the solution path identifies a Transformer weight family with various sparsity levels, offering greater flexibility and customization. In this paper, we introduce such an effective pruning method, termed SPP (Solution Path Pruning). To achieve effective pruning, we segment the transformers into paired modules, including query-key pairs, value-projection pairs, and sequential linear layers, and apply low-rank compression to these pairs, maintaining the output structure while enabling structural compression within the inner states. Extensive experiments conducted on various well-known transformer backbones have demonstrated the efficacy of SPP.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 34 of its 61 references that are in the map. First posted on arXiv 6 Jan 2025.

Builds on

34 of 61 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  6. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  7. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  8. 2018Model compression via distillation and quantizationcited by 287
  9. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
  10. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
  11. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  12. 2021Patch Slimming for Efficient Vision Transformerscited by 131
22 more
  1. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  2. 2015Global Convergence of ADMM in Nonconvex Nonsmooth Optimizationcited by 98
  3. 2020SCOP: Scientific Control for Reliable Neural Network Pruningcited by 92
  4. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  5. 2021Global Vision Transformer Pruning with Hessian-Aware Saliencycited by 69
  6. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  7. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 66
  8. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 51
  9. 2019Stochastic Mirror Descent on Overparameterized Nonlinear Models: Convergence, Implicit Regularization, and Generalizationcited by 40
  10. 2021A Unified Pruning Framework for Vision Transformerscited by 40
  11. 2018Global Convergence of Block Coordinate Descent in Deep Learningcited by 38
  12. 2019Revisiting Knowledge Distillation via Label Smoothing Regularizationcited by 29
  13. 2020MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillationcited by 18
  14. 2021A Bregman Learning Framework for Sparse Neural Networkscited by 15
  15. 2024The Need for Speed: Pruning Transformers with One Recipecited by 15
  16. 2014Sparse Recovery via Differential Inclusionscited by 11
  17. 2023GOHSP: A Unified Framework of Graph and Optimization-based Heterogeneous Structured Pruning for Vision Transformercited by 10
  18. 2024UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformercited by 9
  19. 2017Choose your path wisely: gradient descent in a Bregman distance frameworkcited by 8
  20. 2019Exploring Structural Sparsity of Deep Networks via Inverse Scale Spacescited by 7
  21. 2020DessiLBI: Exploring Structural Sparsity of Deep Networks via Differential Inclusion Pathscited by 3
  22. 2018A Unified Dynamic Approach to Sparse Model Selectioncited by 2

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMscited by 0
  2. 2025UniPruning: Unifying Local Metric and Global Feedback for Scalable Sparse LLMscited by 0