# Low-Resource Machine Translation Training Curriculum Fit for Low-Resource Languages

arXiv: https://arxiv.org/abs/2103.13272 · HTML: https://mlsanity.com/paper/2103.13272/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 70 references that are in the map.
- First posted on arXiv 24 Mar 2021.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages](https://mlsanity.com/paper/2010.02353/) (2020)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://mlsanity.com/paper/2010.12858/) (2020)
- [The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MT](https://mlsanity.com/paper/2010.06354/) (2020)
- [Learning and Evaluating General Linguistic Intelligence](https://mlsanity.com/paper/1901.11373/) (2019)
- [CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP](https://mlsanity.com/paper/2006.06402/) (2020)
- [The Description Length of Deep Learning Models](https://mlsanity.com/paper/1802.07044/) (2018)
- [The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research](https://mlsanity.com/paper/2010.15581/) (2020)
- [Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation](https://mlsanity.com/paper/1708.09803/) (2017)
- [Cross-lingual Retrieval for Iterative Self-Supervised Training](https://mlsanity.com/paper/2006.09526/) (2020)
- [Trivial Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1809.00357/) (2018)
- [Code-switching pre-training for neural machine translation](https://mlsanity.com/paper/2009.08088/) (2020)
- [Learning Language Representations for Typology Prediction](https://mlsanity.com/paper/1707.09569/) (2017)
- [Effective Parallel Corpus Mining using Bilingual Sentence Embeddings](https://mlsanity.com/paper/1807.11906/) (2018)
- [When Does Unsupervised Machine Translation Work?](https://mlsanity.com/paper/2004.05516/) (2020)

## Built on by (most cited first, 3 of 3)

- [Investigating Lexical Replacements for Arabic-English Code-Switched Data Augmentation](https://mlsanity.com/paper/2205.12649/) (2022)
- [Mitigating Translationese in Low-resource Languages: The Storyboard Approach](https://mlsanity.com/paper/2407.10152/) (2024)
- [Replicable Benchmarking of Neural Machine Translation (NMT) on Low-Resource Local Languages in Indonesia](https://mlsanity.com/paper/2311.00998/) (2023)

## Abstract (arXiv)

We conduct an empirical study of neural machine translation (NMT) for truly low-resource languages, and propose a training curriculum fit for cases when both parallel training data and compute resource are lacking, reflecting the reality of most of the world's languages and the researchers working on these languages. Previously, unsupervised NMT, which employs back-translation (BT) and auto-encoding (AE) tasks has been shown barren for low-resource languages. We demonstrate that leveraging comparable data and code-switching as weak supervision, combined with BT and AE objectives, result in remarkable improvements for low-resource languages even when using only modest compute resources. The training curriculum proposed in this work achieves BLEU scores that improve over supervised NMT trained on the same backbone architecture by +12.2 BLEU for English to Gujarati and +3.7 BLEU for English to Kazakh, showcasing the potential of weakly-supervised NMT for the low-resource languages. When trained on supervised data, our training curriculum achieves a new state-of-the-art result on the Somali dataset (BLEU of 29.3 for Somali to English). We also observe that adding more time and GPUs to training can further improve performance, which underscores the importance of reporting compute resource usage in MT research.