mlsanityData to 7 Oct 2026

Reconstruct the Pruned Model without Any Retraining

arXiv 2407.13331 ↗First posted 18 Jul 2024cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Structured pruning is a promising hardware-friendly compression technique for large language models (LLMs), which is expected to be retraining-free to avoid the enormous retraining cost. This retraining-free paradigm involves (1) pruning criteria to define the architecture and (2) distortion reconstruction to restore performance. However, existing methods often emphasize pruning criteria while using reconstruction techniques that are specific to certain modules or criteria, resulting in limited generalizability. To address this, we introduce the Linear Interpolation-based Adaptive Reconstruction (LIAR) framework, which is both efficient and effective. LIAR does not require back-propagation or retraining and is compatible with various pruning criteria and modules. By applying linear interpolation to the preserved weights, LIAR minimizes reconstruction error and effectively reconstructs the pruned output. Our evaluations on benchmarks such as GLUE, SQuAD, WikiText, and common sense reasoning show that LIAR enables a BERT model to maintain 98% accuracy even after removing 50% of its parameters and achieves top performance for LLaMA in just a few minutes.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 46 of its 63 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

46 of 63 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
34 more
  1. 2016Pointer Sentinel Mixture Modelscited by 2,992
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  11. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  13. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  14. 2018Neural Network Acceptability Judgmentscited by 952
  15. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  16. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  17. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  18. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  19. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  20. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  21. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  22. 2019The State of Sparsity in Deep Neural Networkscited by 471
  23. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  24. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  25. 2021Block Pruning For Faster Transformerscited by 176
  26. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  27. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  28. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  29. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  30. 2023Fluctuation-based Adaptive Structured Pruning for Large Language Modelscited by 110
  31. 2020EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inferencecited by 69
  32. 2020Compression of Deep Learning Models for Text: A Surveycited by 64
  33. 2020An Overview of Neural Network Compressioncited by 59
  34. 2023Gradient-Free Structured Pruning with Unlabeled Datacited by 26

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025Combatting Dimensional Collapse in LLM Pre-Training Data via Diversified File Selectioncited by 4
  2. 2025Mining Useful General Data for Low-Resource Domain Adaptationcited by 3
  3. 2025Pruning General Large Language Models into Customized Expert Modelscited by 1
  4. 2025A Free Lunch in LLM Compression: Revisiting Retraining after Pruningcited by 1