mlsanityData to 14 Oct 2026

Gradient-based Intra-attention Pruning on Pre-trained Language Models

arXiv 2212.07634 ↗First posted 15 Dec 2022cs.CLcited by 9

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Pre-trained language models achieve superior performance but are computationally expensive. Techniques such as pruning and knowledge distillation have been developed to reduce their sizes and latencies. In this work, we propose a structured pruning method GRAIN (Gradient-based Intra-attention pruning), which performs task-specific pruning with knowledge distillation and yields highly effective models. Different from common approaches that prune each attention head as a whole, GRAIN inspects and prunes intra-attention structures, which greatly expands the structure search space and enables more flexible models. We also propose a gradient separation strategy that reduces the interference of distillation on pruning for a better combination of the two approaches. Experiments on GLUE, SQuAD, and CoNLL 2003 show that GRAIN notably outperforms other methods, especially in the high sparsity regime, and achieves 6~7× speedups while maintaining 93%~99% performance. Under extreme compression where only 3% transformer weights remain, the pruned model is still competitive compared to larger models.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 39 of its 49 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

39 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation24 Apr

Other references · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  11. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
26 more
  1. 2019Are Sixteen Heads Really Better than One?cited by 816
  2. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  3. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  4. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  5. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  6. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  7. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  8. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  9. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  10. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  11. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
  12. 2019Structured Pruning of Large Language Modelscited by 203
  13. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 185
  14. 2021Block Pruning For Faster Transformerscited by 177
  15. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  16. 2020The Hardware Lotterycited by 160
  17. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  18. 2020OCNLI: Original Chinese Natural Language Inferencecited by 86
  19. 2018A Span-Extraction Dataset for Chinese Machine Reading Comprehensioncited by 84
  20. 2018DRCD: a Chinese Machine Reading Comprehension Datasetcited by 57
  21. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  22. 2021AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Modelscited by 31
  23. 2019Structured Pruning of a BERT-based Question Answering Modelcited by 16
  24. 2022TextPruner: A Model Pruning Toolkit for Pre-Trained Language Modelscited by 10
  25. 2022Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiencycited by 3
  26. 2022COST-EFF: Collaborative Optimization of Spatial and Temporal Efficiency with Slenderized Multi-exit Language Modelscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3; 2025: 4.

9 papers.

  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  2. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  3. 2023Beyond Size: How Gradients Shape Pruning Decisions in Large Language Modelscited by 30
  4. 2024LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Modelscited by 12
  5. 2025Elastic ViTs from Pretrained Models without Retrainingcited by 4
  6. 2023LayerCollapse: Adaptive compression of neural networkscited by 1
  7. 2025Dynamic Acoustic Model Architecture Optimization in Training for ASRcited by 1
  8. 2025Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptationcited by 1
  9. 2025SPAT: Sensitivity-based Multihead-attention Pruning on Time Series Forecasting Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
  2. 2024FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models
  3. 2024A Training-free Sub-quadratic Cost Transformer Model Serving Framework With Hierarchically Pruned Attention
  4. 2025From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models
  5. 2025Improved Methods for Model Pruning and Knowledge Distillation
  6. 2024Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations
  7. 2025E3-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models
  8. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations