# Understanding the effects of language-specific class imbalance in multilingual fine-tuning

arXiv: https://arxiv.org/abs/2402.13016 · HTML: https://mlsanity.com/paper/2402.13016/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 17 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [A Diagnostic Study of Explainability Techniques for Text Classification](https://mlsanity.com/paper/2009.13295/) (2020)
- [The Multilingual Amazon Reviews Corpus](https://mlsanity.com/paper/2010.02573/) (2020)
- [A Corpus for Multilingual Document Classification in Eight Languages](https://mlsanity.com/paper/1805.09821/) (2018)
- [On the Language-specificity of Multilingual BERT and the Impact of Fine-tuning](https://mlsanity.com/paper/2109.06935/) (2021)

## Built on by (most cited first, 7 of 7)

- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [NExtLong: Toward Effective Long-Context Training without Long Documents](https://mlsanity.com/paper/2501.12766/) (2025)
- [Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model](https://mlsanity.com/paper/2405.19846/) (2024)
- [101 Billion Arabic Words Dataset](https://mlsanity.com/paper/2405.01590/) (2024)
- [Leveraging Zipformer Model for Effective Language Identification in Code-Switched Child-Directed Speech](https://mlsanity.com/paper/2508.09430/) (2025)
- [Retrieval Augmented Classification for Confidential Documents](https://mlsanity.com/paper/2604.08628/) (2026)
- [ARAFA: An LLM-Generated Arabic Fact-Checking Dataset](https://mlsanity.com/paper/2609.25833/) (2026)

## Abstract (arXiv)

We study the effect of one type of imbalance often present in real-life multilingual classification datasets: an uneven distribution of labels across languages. We show evidence that fine-tuning a transformer-based Large Language Model (LLM) on a dataset with this imbalance leads to worse performance, a more pronounced separation of languages in the latent space, and the promotion of uninformative features. We modify the traditional class weighing approach to imbalance by calculating class weights separately for each language and show that this helps mitigate those detrimental effects. These results create awareness of the negative effects of language-specific class imbalance in multilingual fine-tuning and the way in which the model learns to rely on the separation of languages to perform the task.