mlsanityData to 14 Oct 2026

Fast and Effective Weight Update for Pruned Large Language Models

arXiv 2401.02938 ↗First posted 1 Jan 2024cs.CL cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Pruning large language models (LLMs) is a challenging task due to their enormous size. The primary difficulty is fine-tuning the model after pruning, which is needed to recover the lost performance caused by dropping weights. Recent approaches have either ignored fine-tuning entirely, focusing on efficient pruning criteria, or attempted layer-wise weight updates, preserving the behavior of each layer. However, even layer-wise weight updates can be costly for LLMs, and previous works have resorted to various approximations. In our paper, we propose a fast and effective weight update algorithm for pruned layers based on the Alternating Direction Method of Multipliers (ADMM). We further extend it with a simple gradual pruning mask selection and achieve state-of-the-art pruning performance across a wide range of LLMs. Code is available at https://github.com/fmfi-compbio/admm-pruning.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 35 of its 46 references that are in the map. First posted on arXiv 1 Jan 2024.

Builds on

35 of 46 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2016Pointer Sentinel Mixture Modelscited by 3,096
23 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  3. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  4. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  7. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  9. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  10. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  11. 2018Rethinking the Value of Network Pruningcited by 599
  12. 2020What is the State of Neural Network Pruning?cited by 583
  13. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  14. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  15. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 251
  16. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  17. 2018A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multiplierscited by 120
  18. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  19. 2019Adversarial Robustness vs Model Compression, or Both?cited by 98
  20. 2019Model Compression with Adversarial Robustness: A Unified Optimization Frameworkcited by 71
  21. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  22. 2023Intriguing Properties of Quantization at Scalecited by 49
  23. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 2.

11 papers.

  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  2. 2025ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMscited by 11
  3. 2024Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorizationcited by 6
  4. 2025MACKO: Sparse Matrix-Vector Multiplication for Low Sparsitycited by 6
  5. 2025The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMMcited by 5
  6. 2025Addition is almost all you need: Compressing large language models with double binary factorizationcited by 3
  7. 2026ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Modelscited by 3
  8. 2024Dependency-Aware Semi-Structured Sparsity of GLU Variants in Large Language Modelscited by 2
  9. 2025Model Hemorrhage and the Robustness Limits of Large Language Modelscited by 1
  10. 2025A Free Lunch in LLM Compression: Revisiting Retraining after Pruningcited by 1
  11. 2026LEAP: Learnable End-to-End Adaptive Pruning of Large Language Modelscited by 0