mlsanityData to 7 Oct 2026

GMP*: Well-Tuned Gradual Magnitude Pruning Can Outperform Most BERT-Pruning Methods

arXiv 2210.06384 ↗First posted 12 Oct 2022cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

We revisit the performance of the classic gradual magnitude pruning (GMP) baseline for large language models, focusing on the classic BERT benchmark on various popular tasks. Despite existing evidence in the literature that GMP performs poorly, we show that a simple and general variant, which we call GMP*, can match and sometimes outperform more complex state-of-the-art methods. Our results provide a simple yet strong baseline for future work, highlight the importance of parameter tuning for baselines, and even improve the performance of the state-of-the-art second-order pruning method in this setting.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 15 of its 20 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

15 of 20 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  6. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  7. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  8. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  9. 2019The State of Sparsity in Deep Neural Networkscited by 471
  10. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  11. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  12. 2021Block Pruning For Faster Transformerscited by 176
3 more
  1. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  2. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  3. 2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40

Built on by

Cited by year

Citing papers per year: 2023: 9; 2024: 3; 2025: 2.

14 papers.

Sort:

  1. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  2. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  3. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  4. 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 46
  5. 2023Scaling Laws for Sparsely-Connected Foundation Modelscited by 41
  6. 2024Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compressioncited by 32
  7. 2024Pruning Large Language Models with Semi-Structural Adaptive Sparse Trainingcited by 20
  8. 2023VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Corescited by 13
  9. 2023Bias in Pruned Vision Models: In-Depth Analysis and Countermeasurescited by 11
  10. 2023Accurate Neural Network Pruning Requires Rethinking Sparse Optimizationcited by 10
  11. 2025SAFE: Finding Sparse and Flat Minima to Improve Pruningcited by 5
  12. 2023oBERTa: Improving Sparse Transfer Learning via improved initialization, distillation, and pruning regimescited by 1
2 more
  1. 2025CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Modelscited by 1
  2. 2023How Does Pruning Impact Long-Tailed Multi-Label Medical Image Classifiers?cited by 0