mlsanityData to 14 Oct 2026

Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Models

arXiv 2406.02924 ↗First posted 5 Jun 2024cs.LG cs.CL cs.NEcited by 40

Not on the map. Why?

Abstract (arXiv)

Despite the remarkable capabilities, Large Language Models (LLMs) face deployment challenges due to their extensive size. Pruning methods drop a subset of weights to accelerate, but many of them require retraining, which is prohibitively expensive and computationally demanding. Recently, post-training pruning approaches introduced novel metrics, enabling the pruning of LLMs without retraining. However, these metrics require the involvement of human experts and tedious trial and error. To efficiently identify superior pruning metrics, we develop an automatic framework for searching symbolic pruning metrics using genetic programming. In particular, we devise an elaborate search space encompassing the existing pruning metrics to discover the potential symbolic pruning metric. We propose an opposing operation simplification strategy to increase the diversity of the population. In this way, Pruner-Zero allows auto-generation of symbolic pruning metrics. Based on the searched results, we explore the correlation between pruning metrics and performance after pruning and summarize some principles. Extensive experiments on LLaMA and LLaMA-2 on language modeling and zero-shot tasks demonstrate that our Pruner-Zero obtains superior performance than SOTA post-training pruning methods. Code at: urlhttps://github.com/pprp/Pruner-Zero.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 70 of its 89 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

70 of 89 references.

References · 70

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
38 more
  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  4. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  6. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  7. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  8. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  9. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  10. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  11. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  12. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  13. 2018Rethinking the Value of Network Pruningcited by 599
  14. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  15. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  16. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  17. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  18. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  19. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  20. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  21. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
  22. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 204
  23. 2023DepGraph: Towards Any Structural Pruningcited by 203
  24. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  25. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  26. 2018Learning Equations for Extrapolation and Controlcited by 156
  27. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 143
  28. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 123
  29. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  30. 2016Extrapolation and learning equationscited by 110
  31. 2020Neural Pruning via Growing Regularizationcited by 97
  32. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  33. 2024Evaluating Quantized Large Language Modelscited by 77
  34. 2022Symbolic Regression is NP-hardcited by 71
  35. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 69
  36. 2024BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocationcited by 45
  37. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  38. 2023The LLM Surgeoncited by 38

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 28; 2026: 5.

40 papers.

Sort:

By area:

  1. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 62
  2. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 50
  3. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 48
  4. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
  5. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Mergingcited by 28
  6. 2025Wanda++: Pruning Large Language Models via Regional Gradientscited by 26
  7. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 16
  8. 2024Beware of Calibration Data for Pruning Large Language Modelscited by 15
  9. 2025Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probingcited by 14
  10. 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Modelscited by 13
  11. 2025ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruningcited by 10
  12. 2025EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Modelscited by 10
28 more
  1. 2025Dynamic Low-Rank Sparse Adaptation for Large Language Modelscited by 10
  2. 2025Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimizationcited by 9
  3. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  4. 2025DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentrationcited by 6
  5. 2025DLP: Dynamic Layerwise Pruning in Large Language Modelscited by 6
  6. 2026On State Reduction in Linear Attentioncited by 6
  7. 2025Progressive Binarization with Semi-Structured Pruning for LLMscited by 5
  8. 2025A Proximal Operator for Inducing 2:4-Sparsitycited by 4
  9. 2025Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Modelscited by 4
  10. 2025DReSS: Data-driven Regularized Structured Streamlining for Large Language Modelscited by 3
  11. 2025AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Modelscited by 3
  12. 2025Beyond One-Size-Fits-All Pruning via Evolutionary Metric Search for Large Language Modelscited by 2
  13. 2025Towards Extreme Pruning of LLMs with Plug-and-Play Mixed Sparsitycited by 2
  14. 2025ACE: Exploring Activation Cosine Similarity and Variance for Accurate and Calibration-Efficient LLM Pruningcited by 2
  15. 2025Enhancing Large Multimodal Models with Adaptive Sparsity and KV Cache Compressioncited by 2
  16. 2025ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorizationcited by 2
  17. 2025W-PCA Based Gradient-Free Proxy for Efficient Search of Lightweight Language Modelscited by 1
  18. 2025SPAP: Structured Pruning via Alternating Optimization and Penalty Methodscited by 1
  19. 2025AccLLM: Accelerating Long-Context LLM Inference Via Algorithm-Hardware Co-Designcited by 1
  20. 2025MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMscited by 1
  21. 2025High-Layer Attention Pruning with Rescalingcited by 1
  22. 2025PermLLM: Learnable Channel Permutation for N:M Sparse Large Language Modelscited by 1
  23. 2026TraceNAS: Zero-shot LLM Pruning via Gradient Trace Correlationcited by 1
  24. 2026GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Modelscited by 1
  25. 2025FastForward Pruning: Efficient LLM Pruning via Single-Step Reinforcement Learningcited by 0
  26. 2025Resting Neurons, Active Insights: Robustifying Activation Sparsity in LLMs via Spontaneitycited by 0
  27. 2026Sparsity Induction for Accurate Post-Training Pruning of Large Language Modelscited by 0
  28. 2026The Sparsity Whisperercited by 0