mlsanityData to 7 Oct 2026

On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning

arXiv 2408.14398 ↗First posted 26 Aug 2024cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Recent advances in large language model (LLM) pruning have shown state-of-the-art (SotA) compression results in post-training and retraining-free settings while maintaining high predictive performance. However, previous research mainly considered calibrating based on English text, despite the multilingual nature of modern LLMs and their frequent use in non-English languages. This analysis paper conducts an in-depth investigation of the performance and internal representation changes associated with pruning multilingual language models for monolingual applications. We present the first comprehensive empirical study, comparing different calibration languages for pruning multilingual models across diverse languages, tasks, models, and SotA pruning techniques. We further analyze the latent subspaces, pruning masks, and individual neurons within pruned models. Our results reveal that while calibration on the target language effectively retains perplexity and yields high signal-to-noise ratios, it does not consistently improve downstream task performance. Further analysis of internal representations at three different levels highlights broader limitations of current pruning approaches: While they effectively preserve dominant information like language-specific features, this is insufficient to counteract the loss of nuanced, language-agnostic features that are crucial for knowledge retention and reasoning.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 43 of its 75 references that are in the map. First posted on arXiv 26 Aug 2024.

Builds on

43 of 75 references.

References · 43

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2023Mistral 7Bcited by 3,065
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  8. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  9. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  10. 2022Mass-Editing Memory in a Transformercited by 976
  11. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  12. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
31 more
  1. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  2. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  3. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  4. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  5. 2023A Survey on Model Compression for Large Language Modelscited by 263
  6. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  7. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  8. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  9. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 151
  10. 2024How do Large Language Models Handle Multilingualism?cited by 148
  11. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
  12. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  13. 2022FP8 Quantization: The Power of the Exponentcited by 87
  14. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  15. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  16. 2021Language Model Evaluation Beyond Perplexitycited by 77
  17. 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 77
  18. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  19. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
  20. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  21. 2023All Languages Matter: On the Multilingual Safety of Large Language Modelscited by 51
  22. 2023On the Impact of Calibration Data in Post-training Quantization and Pruningcited by 51
  23. 2021Revisiting the Primacy of English in Zero-shot Cross-lingual Transfercited by 47
  24. 2023Pruning vs Quantization: Which is Better?cited by 45
  25. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 38
  26. 2023Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuroncited by 25
  27. 2024Discovering Low-rank Subspaces for Language-agnostic Multilingual Representationscited by 18
  28. 2023Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarizationcited by 16
  29. 2024Multilingual Brain Surgeon: Large Language Models Can be Compressed Leaving No Language Behindcited by 10
  30. 2024Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimizationcited by 8
  31. 2024Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Compressing Language Models for Specialized Domainscited by 4
  2. 2026The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answercited by 2
  3. 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1
  4. 2026Relative Kinetic Utility for Reasoning-Aware Structural Pruning in Large Language Modelscited by 0
  5. 2026Averaged Evaluation Masks Capability Trade-Offs: Multi-Source Calibration for High-Sparsity LLM Pruningcited by 0
  6. 2026When Compression Scores Cannot Decide: Information Boundaries for Group-Robust LLM Pruningcited by 0