mlsanityData to 14 Oct 2026

Systematic Weight Evaluation for Pruning Large Language Models: Enhancing Performance and Sustainability

arXiv 2502.17071 ↗First posted 24 Feb 2025cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The exponential growth of large language models (LLMs) like ChatGPT has revolutionized artificial intelligence, offering unprecedented capabilities in natural language processing. However, the extensive computational resources required for training these models have significant environmental implications, including high carbon emissions, energy consumption, and water usage. This research presents a novel approach to LLM pruning, focusing on the systematic evaluation of individual weight importance throughout the training process. By monitoring parameter evolution over time, we propose a method that effectively reduces model size without compromising performance. Extensive experiments with both a scaled-down LLM and a large multimodal model reveal that moderate pruning enhances efficiency and reduces loss, while excessive pruning drastically deteriorates model performance. These findings highlight the critical need for optimized AI models to ensure sustainable development, balancing technological advancement with environmental responsibility.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 26 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

16 of 26 references.

References · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  8. 2020What is the State of Neural Network Pruning?cited by 583
  9. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  10. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 221
  11. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  12. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
4 more
  1. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  2. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  3. 2023Making AI Less "Thirsty": Uncovering and Addressing the Secret Water Footprint of AI Modelscited by 44
  4. 2022Parameter-Efficient Sparsity for Large Language Models Fine-Tuningcited by 27

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimizing Large Language Models: Metrics, Energy Efficiency, and Case Study Insights
  2. 2023Mini-GPTs: Efficient Large Language Models through Contextual Pruning
  3. 2026Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Models
  4. 2025Pruning Laws for Large Language Models
  5. 2024A critical review of methods and challenges in large language models
  6. 2024A Survey of Transformer-based Language Models with Focus on Efficiency
  7. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  8. 2024Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations