mlsanityData to 7 Oct 2026

Movement Pruning: Adaptive Sparsity by Fine-Tuning

arXiv 2005.07683 ↗First posted 15 May 2020cs.CL cs.LGcited by 398

Not on the map. Why?

Cited by 21 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning; however, it is less effective in the transfer learning regime that has become standard for state-of-the-art natural language processing applications. We propose the use of movement pruning, a simple, deterministic first-order weight pruning method that is more adaptive to pretrained model fine-tuning. We give mathematical foundations to the method and compare it to existing zeroth- and first-order pruning methods. Experiments show that when pruning large pretrained language models, movement pruning shows significant improvements in high-sparsity regimes. When combined with distillation, the approach achieves minimal accuracy loss with down to only 3% of the model parameters.

Key facts. Cited by 398 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 47; 2022: 60; 2023: 100; 2024: 75; 2025: 68; 2026: 48.

398 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  5. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  6. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  7. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  8. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  9. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  10. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  11. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  12. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
38 more
  1. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  2. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  3. 2023DepGraph: Towards Any Structural Pruningcited by 202
  4. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  5. 2023Structural Pruning for Diffusion Modelscited by 193
  6. 2021Block Pruning For Faster Transformerscited by 176
  7. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  8. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  9. 2023Speculative Decoding with Big Little Decodercited by 163
  10. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  11. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  12. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  13. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  14. 2021Learned Token Pruning for Transformerscited by 138
  15. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  16. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  17. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  18. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  19. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 100
  20. 2021Learning Compact Metrics for MTcited by 93
  21. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  22. 2023Modular Deep Learningcited by 82
  23. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  24. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  25. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  26. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 75
  27. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  28. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  29. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 66
  30. 2021Low-Complexity Probing via Finding Subnetworkscited by 63
  31. 2022Client Selection in Federated Learning: Principles, Challenges, and Opportunitiescited by 62
  32. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  33. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  34. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  35. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  36. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  37. 2023LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuningcited by 53
  38. 2021A Survey on Green Deep Learningcited by 52

50 most cited of 398 All 398 (JSON)