mlsanityData to 14 Oct 2026

ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models

arXiv 2406.07831 ↗First posted 12 Jun 2024cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

The impressive performance of Large Language Models (LLMs) across various natural language processing tasks comes at the cost of vast computational resources and storage requirements. One-shot pruning techniques offer a way to alleviate these burdens by removing redundant weights without the need for retraining. Yet, the massive scale of LLMs often forces current pruning approaches to rely on heuristics instead of optimization-based techniques, potentially resulting in suboptimal compression. In this paper, we introduce ALPS, an optimization-based framework that tackles the pruning problem using the operator splitting technique and a preconditioned conjugate gradient-based post-processing step. Our approach incorporates novel techniques to accelerate and theoretically guarantee convergence while leveraging vectorization and GPU parallelism for efficiency. ALPS substantially outperforms state-of-the-art methods in terms of the pruning objective and perplexity reduction, particularly for highly sparse models. On the OPT-30B model with 70% sparsity, ALPS achieves a 13% reduction in test perplexity on the WikiText dataset and a 19% improvement in zero-shot benchmark performance compared to existing methods.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 58 of its 72 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

58 of 72 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
38 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,512
  2. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  7. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  8. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  9. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  10. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  11. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  12. 2018Rethinking the Value of Network Pruningcited by 599
  13. 2020What is the State of Neural Network Pruning?cited by 583
  14. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  15. 2019Importance Estimation for Neural Network Pruningcited by 541
  16. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  17. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  18. 2019The State of Sparsity in Deep Neural Networkscited by 476
  19. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  20. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 431
  21. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  22. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  23. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  24. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  25. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  26. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 221
  27. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 213
  28. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
  29. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 151
  30. 2018A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multiplierscited by 120
  31. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  32. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  33. 2015Global Convergence of ADMM in Nonconvex Nonsmooth Optimizationcited by 98
  34. 2019Adversarial Robustness vs Model Compression, or Both?cited by 98
  35. 2015Fast ConvNets Using Group-wise Brain Damagecited by 82
  36. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  37. 2014Convergence Analysis of Alternating Direction Method of Multipliers for a Family of Nonconvex Problemscited by 64
  38. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 12; 2026: 6.

19 papers.

Sort:

By area:

  1. 2024Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Frameworkcited by 11
  2. 2025Dynamic Low-Rank Sparse Adaptation for Large Language Modelscited by 10
  3. 2025SAFE: Finding Sparse and Flat Minima to Improve Pruningcited by 5
  4. 2025SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shotcited by 5
  5. 2025The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMMcited by 5
  6. 2025A Proximal Operator for Inducing 2:4-Sparsitycited by 4
  7. 2025T'yr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimizationcited by 4
  8. 2025Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstructioncited by 4
  9. 2025HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMscited by 3
  10. 2025Restoring Pruned Large Language Models via Lost Component Compensationcited by 3
  11. 2025Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Searchcited by 3
  12. 20263BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMscited by 3
7 more
  1. 2026ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Modelscited by 3
  2. 2025SPAP: Structured Pruning via Alternating Optimization and Penalty Methodscited by 1
  3. 2026NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Modelscited by 1
  4. 2026OBC-Prune: Outcome-Based Calibration for Large Reasoning Model Pruningcited by 1
  5. 2025TSENOR: Highly-Efficient Algorithm for Finding Transposable N:M Sparse Maskscited by 0
  6. 2026The Sparsity Whisperercited by 0
  7. 2026Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Toocited by 0