mlsanityData to 14 Oct 2026

Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models

arXiv 2308.03449 ↗First posted 7 Aug 2023cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Given a pretrained encoder-based language model, how can we accurately compress it without retraining? Retraining-free structured pruning algorithms are crucial in pretrained language model compression due to their significantly reduced pruning cost and capability to prune large language models. However, existing retraining-free algorithms encounter severe accuracy degradation, as they fail to handle pruning errors, especially at high compression rates. In this paper, we propose K-prune (Knowledge-preserving pruning), an accurate retraining-free structured pruning algorithm for pretrained encoder-based language models. K-prune focuses on preserving the useful knowledge of the pretrained model to minimize pruning errors through a carefully designed iterative pruning process composed of knowledge measurement, knowledge-preserving mask search, and knowledge-preserving weight-tuning. As a result, K-prune shows significant accuracy improvements up to 58.02%p higher F1 score compared to existing retraining-free pruning algorithms under a high compression rate of 80% on the SQuAD benchmark without any retraining process.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 47 of its 67 references that are in the map. First posted on arXiv 7 Aug 2023.

Builds on

47 of 67 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2016Layer Normalizationcited by 5,669
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2016Pointer Sentinel Mixture Modelscited by 3,096
35 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  7. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  9. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  10. 2019Are Sixteen Heads Really Better than One?cited by 816
  11. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  12. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  13. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  14. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  15. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  16. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  17. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  18. 2019Structured Pruning of Large Language Modelscited by 203
  19. 2021Block Pruning For Faster Transformerscited by 177
  20. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  21. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  22. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153
  23. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  24. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  25. 2020Densely Guided Knowledge Distillation using Multiple Teacher Assistantscited by 83
  26. 2020Multi-Head Attention: Collaborate Instead of Concatenatecited by 60
  27. 2023Gradient-Free Structured Pruning with Unlabeled Datacited by 26
  28. 2020Neuron Merging: Compensating for Pruned Neuronscited by 24
  29. 2021Post-training deep neural network pruning via layer-wise calibrationcited by 24
  30. 2020Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Priorcited by 23
  31. 2022AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Modelscited by 23
  32. 2021RED : Looking for Redundancies for Data-Free Structured Compression of Deep Neural Networkscited by 19
  33. 2022Exploring Extreme Parameter Compression for Pre-trained Language Modelscited by 15
  34. 2022Understanding and Improving Knowledge Distillation for Quantization-Aware Training of Large Transformer Encoderscited by 11
  35. 2021LEAP: Learnable Pruning for Transformer-based Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 4.

11 papers.

  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  2. 2025Zero-shot Quantization: A Comprehensive Surveycited by 8
  3. 2024MoreauPruner: Robust Pruning of Large Language Models against Weight Perturbationscited by 5
  4. 2025Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformerscited by 5
  5. 2024Hyper-Compression: Model Compression via Hyperfunctioncited by 3
  6. 2026Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compressioncited by 3
  7. 2025Accurate Sublayer Pruning for Large Language Models by Exploiting Latency and Tunability Informationcited by 2
  8. 2024Large Language Model Pruningcited by 1
  9. 2026SynQ: Accurate Zero-shot Quantization by Synthesis-aware Fine-tuningcited by 1
  10. 2026DFMU: Data-Frugal Machine Unlearningcited by 0
  11. 2026FOCUS & RePAIR: Mitigating Text Degeneration via Token-Level Guidance for Pruned Large Language Modelscited by 0