# Pruning Multilingual Large Language Models for Multilingual Inference

arXiv: https://arxiv.org/abs/2409.16911 · HTML: https://mlsanity.com/paper/2409.16911/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [Prompting PaLM for Translation: Assessing Strategies and Performance](https://mlsanity.com/paper/2211.09102/) (2022)
- [The Multilingual Amazon Reviews Corpus](https://mlsanity.com/paper/2010.02573/) (2020)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2310.10378/) (2023)
- [Multilingual Alignment of Contextual Word Representations](https://mlsanity.com/paper/2002.03518/) (2020)
- [Do Multilingual Language Models Think Better in English?](https://mlsanity.com/paper/2308.01223/) (2023)
- [Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00699/) (2020)
- [Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models?](https://mlsanity.com/paper/2311.03788/) (2023)
- [Investigating Bias in Multilingual Language Models: Cross-Lingual Transfer of Debiasing Techniques](https://mlsanity.com/paper/2310.10310/) (2023)
- [An Empirical Study of In-context Learning in LLMs for Machine Translation](https://mlsanity.com/paper/2401.12097/) (2024)
- [Bilingual Alignment Pre-Training for Zero-Shot Cross-Lingual Transfer](https://mlsanity.com/paper/2106.01732/) (2021)
- [Cross-lingual Transfer Can Worsen Bias in Sentiment Analysis](https://mlsanity.com/paper/2305.12709/) (2023)

## Built on by (most cited first, 2 of 2)

- [Multilinguality at the Edge: Developing Language Models for the Global South](https://mlsanity.com/paper/2604.21637/) (2026)
- [NLP Privacy Risk Identification in Social Media (NLP-PRISM): A Survey](https://mlsanity.com/paper/2602.15866/) (2026)

## Abstract (arXiv)

Multilingual large language models (MLLMs), trained on multilingual balanced data, demonstrate better zero-shot learning performance in non-English languages compared to large language models trained on English-dominant data. However, the disparity in performance between English and non-English languages remains a challenge yet to be fully addressed. A distinctive characteristic of MLLMs is their high-quality translation capabilities, indicating an acquired proficiency in aligning between languages. This study explores how to enhance the zero-shot performance of MLLMs in non-English languages by leveraging their alignment capability between English and non-English languages. To achieve this, we first analyze the behavior of MLLMs when performing translation and reveal that there are large magnitude features that play a critical role in the translation process. Inspired by these findings, we retain the weights associated with operations involving the large magnitude features and prune other weights to force MLLMs to rely on these features for tasks beyond translation. We empirically demonstrate that this pruning strategy can enhance the MLLMs' performance in non-English language.