# SparseLLM: Towards Global Pruning for Pre-trained Language Models

arXiv: https://arxiv.org/abs/2402.17946 · HTML: https://mlsanity.com/paper/2402.17946/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 28 Feb 2024.

## Builds on (most cited first, 28 of 28)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [A Fast Post-Training Pruning Framework for Transformers](https://mlsanity.com/paper/2204.09656/) (2022)
- [LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation](https://mlsanity.com/paper/2306.11222/) (2023)
- [Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks](https://mlsanity.com/paper/2102.08124/) (2021)
- [Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models](https://mlsanity.com/paper/2401.00625/) (2024)
- [GPT Takes the Bar Exam](https://mlsanity.com/paper/2212.14402/) (2022)
- [A Survey on Model Compression and Acceleration for Pretrained Language Models](https://mlsanity.com/paper/2202.07105/) (2022)
- [AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers](https://mlsanity.com/paper/2302.14705/) (2023)
- [ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models](https://mlsanity.com/paper/2310.02998/) (2023)
- [FFSplit: Split Feed-Forward Network For Optimizing Accuracy-Efficiency Trade-off in Language Model Inference](https://mlsanity.com/paper/2401.04044/) (2024)

## Built on by (most cited first, 23 of 23)

- [MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE](https://mlsanity.com/paper/2507.00390/) (2025)
- [ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs](https://mlsanity.com/paper/2502.00258/) (2025)
- [Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models](https://mlsanity.com/paper/2505.01731/) (2025)
- [LatentLLM: Attention-Aware Joint Tensor Compression](https://mlsanity.com/paper/2505.18413/) (2025)
- [The Right to be Forgotten in Pruning: Unveil Machine Unlearning on Sparse Models](https://mlsanity.com/paper/2507.18725/) (2025)
- [A Survey of Transformer-based Language Models with Focus on Efficiency](https://mlsanity.com/paper/2406.16893/) (2024)
- [From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models](https://mlsanity.com/paper/2510.18030/) (2025)
- [The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM](https://mlsanity.com/paper/2510.01650/) (2025)
- [μ-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts](https://mlsanity.com/paper/2505.18451/) (2025)
- [NeuroAgent: LLM Agents for Multimodal Neuroimaging Analysis and Research](https://mlsanity.com/paper/2605.06584/) (2026)
- [Beyond One-Size-Fits-All Pruning via Evolutionary Metric Search for Large Language Models](https://mlsanity.com/paper/2502.10735/) (2025)
- [Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization](https://mlsanity.com/paper/2502.15763/) (2025)
- [PermLLM: Learnable Channel Permutation for N:M Sparse Large Language Models](https://mlsanity.com/paper/2510.10136/) (2025)
- [Frustratingly Easy Task-aware Pruning for Large Language Models](https://mlsanity.com/paper/2510.22489/) (2025)
- [TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly](https://mlsanity.com/paper/2603.19296/) (2026)
- [Scalable Cross-Facility Federated Learning for Scientific Foundation Models on Multiple Supercomputers](https://mlsanity.com/paper/2603.19544/) (2026)
- [TuneComp: Joint Fine-tuning and Compression for Large Foundation Models](https://mlsanity.com/paper/2505.21835/) (2025)
- [EGGS-PTP: An Expander-Graph Guided Structured Post-training Pruning Method for Large Language Models](https://mlsanity.com/paper/2508.09471/) (2025)
- [UniPruning: Unifying Local Metric and Global Feedback for Scalable Sparse LLMs](https://mlsanity.com/paper/2510.03291/) (2025)
- [Big2Small: A Unifying Neural Network Framework for Model Compression](https://mlsanity.com/paper/2603.29768/) (2026)
- [Relative Kinetic Utility for Reasoning-Aware Structural Pruning in Large Language Models](https://mlsanity.com/paper/2605.09008/) (2026)
- [EinSort: Sorting is All We Need for Tensorizing LLM](https://mlsanity.com/paper/2606.08565/) (2026)
- [At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference](https://mlsanity.com/paper/2607.25504/) (2026)

## Abstract (arXiv)

The transformative impact of large language models (LLMs) like LLaMA and GPT on natural language processing is countered by their prohibitive computational demands. Pruning has emerged as a pivotal compression strategy, introducing sparsity to enhance both memory and computational efficiency. Yet, traditional global pruning is impractical for LLMs due to scalability issues, while local pruning, despite its efficiency, leads to suboptimal solutions. Addressing these challenges, we propose SparseLLM, a novel framework that redefines the global pruning process into manageable, coordinated subproblems, allowing for resource-efficient optimization with global optimality. SparseLLM's approach, which conceptualizes LLMs as a chain of modular functions and leverages auxiliary variables for problem decomposition, not only facilitates a pragmatic application on LLMs but also demonstrates significant performance improvements, particularly in high-sparsity regimes where it surpasses current state-of-the-art methods.