# Should we Stop Training More Monolingual Models, and Simply Use Machine Translation Instead?

arXiv: https://arxiv.org/abs/2104.10441 · HTML: https://mlsanity.com/paper/2104.10441/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 21 Apr 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [A Survey Of Cross-lingual Word Embedding Models](https://mlsanity.com/paper/1706.04902/) (2017)
- [Zero-Shot Cross-Lingual Transfer with Meta Learning](https://mlsanity.com/paper/2003.02739/) (2020)

## Built on by (most cited first, 17 of 17)

- [mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset](https://mlsanity.com/paper/2108.13897/) (2021)
- [Revisiting Machine Translation for Cross-lingual Classification](https://mlsanity.com/paper/2305.14240/) (2023)
- [Modelling Latent Translations for Cross-Lingual Transfer](https://mlsanity.com/paper/2107.11353/) (2021)
- [Conversational Feedback in Scripted versus Spontaneous Dialogues: A Comparative Analysis](https://mlsanity.com/paper/2309.15656/) (2023)
- [Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages](https://mlsanity.com/paper/2406.12739/) (2024)
- [A cost-benefit analysis of cross-lingual transfer methods](https://mlsanity.com/paper/2105.06813/) (2021)
- [To Translate or Not to Translate: A Systematic Investigation of Translation-Based Cross-Lingual Transfer to Low-Resource Languages](https://mlsanity.com/paper/2311.09404/) (2023)
- [GrEmLIn: A Repository of Green Baseline Embeddings for 87 Low-Resource Languages Injected with Multilingual Graph Knowledge](https://mlsanity.com/paper/2409.18193/) (2024)
- [Dr.Copilot: A Multi-Agent Prompt Optimized Assistant for Improving Patient-Doctor Communication in Romanian](https://mlsanity.com/paper/2507.11299/) (2025)
- [Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languages](https://mlsanity.com/paper/2502.10140/) (2025)
- [Cross-Lingual Knowledge Transfer for Clinical Phenotyping](https://mlsanity.com/paper/2208.01912/) (2022)
- [Generalized Funnelling: Ensemble Learning and Heterogeneous Document Embeddings for Cross-Lingual Text Classification](https://mlsanity.com/paper/2110.14764/) (2021)
- [MASIVE: Open-Ended Affective State Identification in English and Spanish](https://mlsanity.com/paper/2407.12196/) (2024)
- [Code-switching in text and speech challenges information-theoretic speaker design](https://mlsanity.com/paper/2408.04596/) (2024)
- [FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Models](https://mlsanity.com/paper/2512.13330/) (2025)
- [Evaluating Cross-Lingual Classification Approaches Enabling Topic Discovery for Multilingual Social Media Data](https://mlsanity.com/paper/2602.17051/) (2026)
- [TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling](https://mlsanity.com/paper/2609.26347/) (2026)

## Abstract (arXiv)

Most work in NLP makes the assumption that it is desirable to develop solutions in the native language in question. There is consequently a strong trend towards building native language models even for low-resource languages. This paper questions this development, and explores the idea of simply translating the data into English, thereby enabling the use of pretrained, and large-scale, English language models. We demonstrate empirically that a large English language model coupled with modern machine translation outperforms native language models in most Scandinavian languages. The exception to this is Finnish, which we assume is due to inferior translation quality. Our results suggest that machine translation is a mature technology, which raises a serious counter-argument for training native language models for low-resource languages. This paper therefore strives to make a provocative but important point. As English language models are improving at an unprecedented pace, which in turn improves machine translation, it is from an empirical and environmental stand-point more effective to translate data from low-resource languages into English, than to build language models for such languages.