# MultiFiT: Efficient Multi-lingual Language Model Fine-tuning

arXiv: https://arxiv.org/abs/1909.04761 · HTML: https://mlsanity.com/paper/1909.04761/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2019.

## Built on by (most cited first, 30 of 31)

- [MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://mlsanity.com/paper/2109.00904/) (2021)
- [Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalization](https://mlsanity.com/paper/2601.12993/) (2026)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Improved Multi-label Classification under Temporal Concept Drift: Rethinking Group-Robust Algorithms in a Label-Wise Setting](https://mlsanity.com/paper/2203.07856/) (2022)
- [Taxi1500: A Multilingual Dataset for Text Classification in 1500 Languages](https://mlsanity.com/paper/2305.08487/) (2023)
- [Analysing The Impact Of Linguistic Features On Cross-Lingual Transfer](https://mlsanity.com/paper/2105.05975/) (2021)
- [Everything Is All It Takes: A Multipronged Strategy for Zero-Shot Cross-Lingual Information Extraction](https://mlsanity.com/paper/2109.06798/) (2021)
- [Multilingual and cross-lingual document classification: A meta-learning approach](https://mlsanity.com/paper/2101.11302/) (2021)
- [Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning](https://mlsanity.com/paper/2411.18571/) (2024)
- [TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese](https://mlsanity.com/paper/2401.16640/) (2024)
- [Can Character-based Language Models Improve Downstream Task Performance in Low-Resource and Noisy Language Scenarios?](https://mlsanity.com/paper/2110.13658/) (2021)
- [BayLing 2: A Multilingual Large Language Model with Efficient Language Alignment](https://mlsanity.com/paper/2411.16300/) (2024)
- [Fake News Detection System using XLNet model with Topic Distributions: CONSTRAINT@AAAI2021 Shared Task](https://mlsanity.com/paper/2101.11425/) (2021)
- [Lightweight Cross-Lingual Sentence Representation Learning](https://mlsanity.com/paper/2105.13856/) (2021)
- [Rumour Detection via Zero-shot Cross-lingual Transfer Learning](https://mlsanity.com/paper/2109.12773/) (2021)
- [LegaLMFiT: Efficient Short Legal Text Classification with LSTM Language Model Pre-Training](https://mlsanity.com/paper/2109.00993/) (2021)
- [Multilingual Transformer Encoders: a Word-Level Task-Agnostic Evaluation](https://mlsanity.com/paper/2207.09076/) (2022)
- [Cross-lingual Transfer Can Worsen Bias in Sentiment Analysis](https://mlsanity.com/paper/2305.12709/) (2023)
- [Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations](https://mlsanity.com/paper/2311.03999/) (2023)
- [A Multilingual Bag-of-Entities Model for Zero-Shot Cross-Lingual Text Classification](https://mlsanity.com/paper/2110.07792/) (2021)
- [Multilingual Multiword Expression Identification Using Lateral Inhibition and Domain Adaptation](https://mlsanity.com/paper/2306.10419/) (2023)
- [Improving Training Efficiency and Reducing Maintenance Costs via Language Specific Model Merging](https://mlsanity.com/paper/2601.16127/) (2026)
- [Improved Multilingual Language Model Pretraining for Social Media Text via Translation Pair Prediction](https://mlsanity.com/paper/2110.10318/) (2021)
- [Generalized Funnelling: Ensemble Learning and Heterogeneous Document Embeddings for Cross-Lingual Text Classification](https://mlsanity.com/paper/2110.14764/) (2021)
- [Improved Customer Transaction Classification using Semi-Supervised Knowledge Distillation](https://mlsanity.com/paper/2102.07635/) (2021)
- [Priberam Labs at the NTCIR-15 SHINRA2020-ML: Classification Task](https://mlsanity.com/paper/2105.05605/) (2021)
- [Gradient Sparsification For Masked Fine-Tuning of Transformers](https://mlsanity.com/paper/2307.10098/) (2023)
- [Zero-shot Cross-lingual Transfer without Parallel Corpus](https://mlsanity.com/paper/2310.04726/) (2023)
- [Effective vocabulary expansion of multilingual language models for extremely low-resource languages](https://mlsanity.com/paper/2602.09388/) (2026)
- [MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languages](https://mlsanity.com/paper/2603.20732/) (2026)

## Abstract (arXiv)

Pretrained language models are promising particularly for low-resource languages as they only require unlabelled data. However, training existing models requires huge amounts of compute, while pretrained cross-lingual models often underperform on low-resource languages. We propose Multi-lingual language model Fine-Tuning (MultiFiT) to enable practitioners to train and fine-tune language models efficiently in their own language. In addition, we propose a zero-shot method using an existing pretrained cross-lingual model. We evaluate our methods on two widely used cross-lingual classification datasets where they outperform models pretrained on orders of magnitude more data and compute. We release all models and code.