# Wanda++: Pruning Large Language Models via Regional Gradients

arXiv: https://arxiv.org/abs/2503.04992 · HTML: https://mlsanity.com/paper/2503.04992/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Placed in

- Efficient LLM inference and serving / [Early exit and edge inference](https://mlsanity.com/inference/early-exit-and-edge-inference/)

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [Language model compression with weighted low-rank factorization](https://mlsanity.com/paper/2207.00112/) (2022)
- [FP8 Quantization: The Power of the Exponent](https://mlsanity.com/paper/2208.09225/) (2022)
- [Revisiting Locally Supervised Learning: an Alternative to End-to-end Training](https://mlsanity.com/paper/2101.10832/) (2021)
- [Structural Pruning via Latency-Saliency Knapsack](https://mlsanity.com/paper/2210.06659/) (2022)
- [Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Models](https://mlsanity.com/paper/2406.02924/) (2024)
- [Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models](https://mlsanity.com/paper/2311.04902/) (2023)
- [HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers](https://mlsanity.com/paper/2302.09632/) (2023)
- [Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks](https://mlsanity.com/paper/2401.02731/) (2024)
- [BEBERT: Efficient and Robust Binary Ensemble BERT](https://mlsanity.com/paper/2210.15976/) (2022)
- [How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective](https://mlsanity.com/paper/2306.05857/) (2023)
- [Efficient N:M Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Design](https://mlsanity.com/paper/2309.13015/) (2023)

## Built on by (most cited first, 26 of 26)

- [SepPrune: Structured Pruning for Efficient Deep Speech Separation](https://mlsanity.com/paper/2505.12079/) (2025)
- [Block-Sparse Global Attention for Efficient Multi-View Geometry Transformers](https://mlsanity.com/paper/2509.07120/) (2025)
- [Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning](https://mlsanity.com/paper/2601.06787/) (2026)
- [From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models](https://mlsanity.com/paper/2510.18030/) (2025)
- [On State Reduction in Linear Attention](https://mlsanity.com/paper/2602.04852/) (2026)
- [FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compression](https://mlsanity.com/paper/2505.23966/) (2025)
- [ACE: Exploring Activation Cosine Similarity and Variance for Accurate and Calibration-Efficient LLM Pruning](https://mlsanity.com/paper/2505.21987/) (2025)
- [LLMs can Compress LLMs: Adaptive Pruning by Agents](https://mlsanity.com/paper/2601.09694/) (2026)
- [Sink-Aware Pruning for Diffusion Language Models](https://mlsanity.com/paper/2602.17664/) (2026)
- [GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models](https://mlsanity.com/paper/2604.19398/) (2026)
- [Weight Pruning Amplifies Bias: A Multi-Method Study of Compressed LLMs for Edge AI](https://mlsanity.com/paper/2605.08137/) (2026)
- [Frustratingly Easy Task-aware Pruning for Large Language Models](https://mlsanity.com/paper/2510.22489/) (2025)
- [TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Training](https://mlsanity.com/paper/2511.03983/) (2025)
- [TraceNAS: Zero-shot LLM Pruning via Gradient Trace Correlation](https://mlsanity.com/paper/2602.02891/) (2026)
- [GRAIL: Post-hoc Compensation by Linear Reconstruction for Compressed Networks](https://mlsanity.com/paper/2602.23795/) (2026)
- [Frequency Matters: Fast Model-Agnostic Data Curation for Pruning and Quantization](https://mlsanity.com/paper/2603.16105/) (2026)
- [Small LLMs: Pruning vs. Training from Scratch](https://mlsanity.com/paper/2606.14150/) (2026)
- [The Asymmetric Harms of LLM Compression](https://mlsanity.com/paper/2608.19670/) (2026)
- [Can abstract concepts from LLM improve SLM performance?](https://mlsanity.com/paper/2512.19069/) (2025)
- [UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization, and Distillation](https://mlsanity.com/paper/2602.09130/) (2026)
- [Alternating Gradient Flow Utility: A Unified Metric for Structural Pruning and Dynamic Routing in Deep Networks](https://mlsanity.com/paper/2603.12354/) (2026)
- [Structural Sensitivity in Compressed Transformers: Relative Error Propagation and Layer Removal](https://mlsanity.com/paper/2603.20991/) (2026)
- [Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMs](https://mlsanity.com/paper/2607.10803/) (2026)
- [TriSP: Tri-Signal Structured Pruning for Large Language Models](https://mlsanity.com/paper/2607.22587/) (2026)
- [At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference](https://mlsanity.com/paper/2607.25504/) (2026)
- [Theoretical Guarantees for One-Shot Magnitude Pruning and Compute-Adaptive Early Exit](https://mlsanity.com/paper/2609.12337/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) pruning seeks to remove unimportant weights for inference speedup with minimal accuracy impact. However, existing methods often suffer from accuracy degradation without full-model sparsity-aware fine-tuning. This paper presents Wanda++, a novel pruning framework that outperforms the state-of-the-art methods by utilizing decoder-block-level regional gradients. Specifically, Wanda++ improves the pruning score with regional gradients for the first time and proposes an efficient regional optimization method to minimize pruning-induced output discrepancies between the dense and sparse decoder output. Notably, Wanda++ improves perplexity by up to 32% over Wanda in the language modeling task and generalizes effectively to downstream tasks. Moreover, despite updating weights with regional optimization, Wanda++ remains orthogonal to sparsity-aware fine-tuning, further reducing perplexity with LoRA in great extend. Our approach is lightweight, pruning a 7B LLaMA model in under 10 minutes on a single H100 GPU.