F-WANDA: Fisher-Reweighted Post-Training Pruning for Sustainable Deployment of Large Language Models
Not on the map. Why?
Abstract (arXiv)
One-shot post-training pruning is the most energy-frugal compression strategy for largelanguage models (LLMs), yet existing approaches trade either quality (WANDA) or compute cost (SPARSEGPT). We introduce F-WANDA, a drop-in modification of WANDA that reallocates the per-row keep budget across output neurons in proportion to the empirical Fisher information of the pre-activation. The Fisher signal is collected in a single additional backward pass over the same calibration corpus WANDA already uses; no weights are updated. On LLAMA-2-7B at 50 % unstructured sparsity, F-WANDA attains WikiText-2 perplexity of 6.85, matches WANDA fluency, and improves 5-shot MMLU by +1.6 pp over WANDA and +1.1 pp over SPARSEGPT, while incurring only one-third of SPARSEGPT pruning wall-clock and energy. The headline trade-off is achieved without extra calibration data or fine-tuning, placing F-WANDA on the Pareto frontier of quality versus pruning cost for sustainable LLM compression.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 24 references that are in the map. First posted on arXiv 1 Aug 2026.
Builds on
20 of 24 references.
References · 20
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
- 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
- 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
8 more
- 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
- 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
- 2019Green AIcited by 620
- 2021Carbon Emissions and Large Neural Network Trainingcited by 561
- 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 246
- 2021Accelerating Sparse Deep Neural Networkscited by 213
- 2021Group Fisher Pruning for Practical Network Compressioncited by 130
- 2018Faster gaze prediction with dense networks and Fisher pruningcited by 92
Built on by
Not cited yet.