# Load What You Need: Smaller Versions of Multilingual BERT

arXiv: https://arxiv.org/abs/2010.05609 · HTML: https://mlsanity.com/paper/2010.05609/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2020.

## Built on by (most cited first, 30 of 32)

- [Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning](https://mlsanity.com/paper/2204.06487/) (2022)
- [Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training](https://mlsanity.com/paper/2212.10503/) (2022)
- [An Efficient Multilingual Language Model Compression through Vocabulary Trimming](https://mlsanity.com/paper/2305.15020/) (2023)
- [DziriBERT: a Pre-trained Language Model for the Algerian Dialect](https://mlsanity.com/paper/2109.12346/) (2021)
- [ScandEval: A Benchmark for Scandinavian Natural Language Processing](https://mlsanity.com/paper/2304.00906/) (2023)
- [Intriguing Properties of Compression on Multilingual Models](https://mlsanity.com/paper/2211.02738/) (2022)
- [Methods for Detoxification of Texts for the Russian Language](https://mlsanity.com/paper/2105.09052/) (2021)
- [You Are What You Write: Preserving Privacy in the Era of Large Language Models](https://mlsanity.com/paper/2204.09391/) (2022)
- [TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models](https://mlsanity.com/paper/2203.15996/) (2022)
- [On the Usability of Transformers-based models for a French Question-Answering task](https://mlsanity.com/paper/2207.09150/) (2022)
- [Accelerating Multilingual Language Model for Excessively Tokenized Languages](https://mlsanity.com/paper/2401.10660/) (2024)
- [Accenture at CheckThat! 2021: Interesting claim identification and ranking with contextually sensitive lexical training data augmentation](https://mlsanity.com/paper/2107.05684/) (2021)
- [The futility of STILTs for the classification of lexical borrowings in Spanish](https://mlsanity.com/paper/2109.08607/) (2021)
- [Distilling Efficient Language-Specific Models for Cross-Lingual Transfer](https://mlsanity.com/paper/2306.01709/) (2023)
- [Speculative Decoding with a Speculative Vocabulary](https://mlsanity.com/paper/2602.13836/) (2026)
- [Multi-stage Training of Bilingual Islamic LLM for Neural Passage Retrieval](https://mlsanity.com/paper/2501.10175/) (2025)
- [BaldWhisper: Faster Whisper with Head Shearing and Layer Merging](https://mlsanity.com/paper/2510.08599/) (2025)
- [Benchmarking Transformers-based models on French Spoken Language Understanding tasks](https://mlsanity.com/paper/2207.09152/) (2022)
- [The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics](https://mlsanity.com/paper/2311.09709/) (2023)
- [Building an Efficient Multilingual Non-Profit IR System for the Islamic Domain Leveraging Multiprocessing Design in Rust](https://mlsanity.com/paper/2411.06151/) (2024)
- [Prune or Retrain: Optimizing the Vocabulary of Multilingual Models for Estonian](https://mlsanity.com/paper/2501.02631/) (2025)
- [Auto-tagging of Short Conversational Sentences using Transformer Methods](https://mlsanity.com/paper/2106.01735/) (2021)
- [Vocabulary-level Memory Efficiency for Language Model Fine-tuning](https://mlsanity.com/paper/2309.08708/) (2023)
- [Low Resource Summarization using Pre-trained Language Models](https://mlsanity.com/paper/2310.02790/) (2023)
- [ZusammenQA: Data Augmentation with Specialized Models for Cross-lingual Open-retrieval Question Answering System](https://mlsanity.com/paper/2205.14981/) (2022)
- [Efficient and Versatile Model for Multilingual Information Retrieval of Islamic Text: Development and Deployment in Real-World Scenarios](https://mlsanity.com/paper/2509.15380/) (2025)
- [idT5: Indonesian Version of Multilingual T5 Transformer](https://mlsanity.com/paper/2302.00856/) (2023)
- [Assessing Linguistic Generalisation in Language Models: A Dataset for Brazilian Portuguese](https://mlsanity.com/paper/2305.14070/) (2023)
- [On Multilingual Encoder Language Model Compression for Low-Resource Languages](https://mlsanity.com/paper/2505.16956/) (2025)
- [EmoBench-UA: A Benchmark Dataset for Emotion Detection in Ukrainian](https://mlsanity.com/paper/2505.23297/) (2025)

## Abstract (arXiv)

Pre-trained Transformer-based models are achieving state-of-the-art results on a variety of Natural Language Processing data sets. However, the size of these models is often a drawback for their deployment in real production applications. In the case of multilingual models, most of the parameters are located in the embeddings layer. Therefore, reducing the vocabulary size should have an important impact on the total number of parameters. In this paper, we propose to generate smaller models that handle fewer number of languages according to the targeted corpora. We present an evaluation of smaller versions of multilingual BERT on the XNLI data set, but we believe that this method may be applied to other multilingual transformers. The obtained results confirm that we can generate smaller models that keep comparable results, while reducing up to 45% of the total number of parameters. We compared our models with DistilmBERT (a distilled version of multilingual BERT) and showed that unlike language reduction, distillation induced a 1.7% to 6% drop in the overall accuracy on the XNLI data set. The presented models and code are publicly available.