# Multilingual Brain Surgeon: Large Language Models Can be Compressed Leaving No Language Behind

arXiv: https://arxiv.org/abs/2404.04748 · HTML: https://mlsanity.com/paper/2404.04748/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 6 Apr 2024.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [Group Fisher Pruning for Practical Network Compression](https://mlsanity.com/paper/2108.00708/) (2021)
- [Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks](https://mlsanity.com/paper/2102.08124/) (2021)
- [OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization](https://mlsanity.com/paper/2304.07493/) (2023)
- [Larger-Scale Transformers for Multilingual Masked Language Modeling](https://mlsanity.com/paper/2105.00572/) (2021)
- [ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation](https://mlsanity.com/paper/2303.08302/) (2023)
- [AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks](https://mlsanity.com/paper/2106.12379/) (2021)
- [How Well Do Sparse Imagenet Models Transfer?](https://mlsanity.com/paper/2111.13445/) (2021)
- [Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs](https://mlsanity.com/paper/2309.05516/) (2023)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [QuantEase: Optimization-based Quantization for Language Models](https://mlsanity.com/paper/2309.01885/) (2023)

## Built on by (most cited first, 11 of 11)

- [Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models](https://mlsanity.com/paper/2410.11718/) (2024)
- [Self-calibration for Language Model Quantization and Pruning](https://mlsanity.com/paper/2410.17170/) (2024)
- [Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization](https://mlsanity.com/paper/2510.10618/) (2025)
- [Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient](https://mlsanity.com/paper/2406.10576/) (2024)
- [On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning](https://mlsanity.com/paper/2408.14398/) (2024)
- [When Language Shapes Thought: Cross-Lingual Transfer of Factual Knowledge in Question Answering](https://mlsanity.com/paper/2505.24409/) (2025)
- [M-Wanda: Improving One-Shot Pruning for Multilingual LLMs](https://mlsanity.com/paper/2505.21171/) (2025)
- [XToM: Exploring the Multilingual Theory of Mind for Large Language Models](https://mlsanity.com/paper/2506.02461/) (2025)
- [Multilinguality at the Edge: Developing Language Models for the Global South](https://mlsanity.com/paper/2604.21637/) (2026)
- [UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization, and Distillation](https://mlsanity.com/paper/2602.09130/) (2026)
- [Lightweight but Linguistically Fair? Quantisation Robustness across Bavarian, German, and English](https://mlsanity.com/paper/2610.18343/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have ushered in a new era in Natural Language Processing, but their massive size demands effective compression techniques for practicality. Although numerous model compression techniques have been investigated, they typically rely on a calibration set that overlooks the multilingual context and results in significant accuracy degradation for low-resource languages. This paper introduces Multilingual Brain Surgeon (MBS), a novel calibration data sampling method for multilingual LLMs compression. MBS overcomes the English-centric limitations of existing methods by sampling calibration data from various languages proportionally to the language distribution of the model training datasets. Our experiments, conducted on the BLOOM multilingual LLM, demonstrate that MBS improves the performance of existing English-centric compression methods, especially for low-resource languages. We also uncover the dynamics of language interaction during compression, revealing that the larger the proportion of a language in the training set and the more similar the language is to the calibration language, the better performance the language retains after compression. In conclusion, MBS presents an innovative approach to compressing multilingual LLMs, addressing the performance disparities and improving the language inclusivity of existing compression techniques.