# AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languages

arXiv: https://arxiv.org/abs/2109.04715 · HTML: https://mlsanity.com/paper/2109.04715/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages](https://mlsanity.com/paper/2010.02353/) (2020)
- [Rapid Adaptation of Neural Machine Translation to New Languages](https://mlsanity.com/paper/1808.04189/) (2018)
- [The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English](https://mlsanity.com/paper/1902.01382/) (2019)
- [Cross-lingual Retrieval for Iterative Self-Supervised Training](https://mlsanity.com/paper/2006.09526/) (2020)
- [Masakhane -- Machine Translation For Africa](https://mlsanity.com/paper/2003.11529/) (2020)
- [Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2105.03953/) (2021)
- [From Speech-to-Speech Translation to Automatic Dubbing](https://mlsanity.com/paper/2001.06785/) (2020)
- [PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining](https://mlsanity.com/paper/2108.01887/) (2021)
- [Benchmarking Neural Machine Translation for Southern African Languages](https://mlsanity.com/paper/1906.10511/) (2019)

## Built on by (most cited first, 24 of 24)

- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages](https://mlsanity.com/paper/2212.05409/) (2022)
- [A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation](https://mlsanity.com/paper/2205.02022/) (2022)
- [Building Machine Translation Systems for the Next Thousand Languages](https://mlsanity.com/paper/2205.03983/) (2022)
- [MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition](https://mlsanity.com/paper/2210.12391/) (2022)
- [IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languages](https://mlsanity.com/paper/2212.10180/) (2022)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [Cheetah: Natural Language Generation for 517 African Languages](https://mlsanity.com/paper/2401.01053/) (2024)
- [The Zeno's Paradox of `Low-Resource' Languages](https://mlsanity.com/paper/2410.20817/) (2024)
- [MMTAfrica: Multilingual Machine Translation for African Languages](https://mlsanity.com/paper/2204.04306/) (2022)
- [Uncertainty-Aware Decoding with Minimum Bayes Risk](https://mlsanity.com/paper/2503.05318/) (2025)
- [TaTa: A Multilingual Table-to-Text Dataset for African Languages](https://mlsanity.com/paper/2211.00142/) (2022)
- [Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning](https://mlsanity.com/paper/2405.19462/) (2024)
- [Where Are We? Evaluating LLM Performance on African Languages](https://mlsanity.com/paper/2502.19582/) (2025)
- [Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models](https://mlsanity.com/paper/2305.12544/) (2023)
- [A Natural Bias for Language Generation Models](https://mlsanity.com/paper/2212.09686/) (2022)
- [Investigating Neural Machine Translation for Low-Resource Languages: Using Bavarian as a Case Study](https://mlsanity.com/paper/2404.08259/) (2024)
- [NaijaRC: A Multi-choice Reading Comprehension Dataset for Nigerian Languages](https://mlsanity.com/paper/2308.09768/) (2023)
- [Voice of a Continent: Mapping Africa's Speech Technology Frontier](https://mlsanity.com/paper/2505.18436/) (2025)
- [ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Model](https://mlsanity.com/paper/2404.02534/) (2024)
- [AfriWOZ: Corpus for Exploiting Cross-Lingual Transferability for Generation of Dialogues in Low-Resource, African Languages](https://mlsanity.com/paper/2204.08083/) (2022)
- [PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generation](https://mlsanity.com/paper/2512.13552/) (2025)
- [The African Languages Lab: A Collaborative Approach to Advancing Low-Resource African NLP](https://mlsanity.com/paper/2510.05644/) (2025)
- [Pretraining Strategies using Monolingual and Parallel Data for Low-Resource Machine Translation](https://mlsanity.com/paper/2510.25116/) (2025)

## Abstract (arXiv)

Reproducible benchmarks are crucial in driving progress of machine translation research. However, existing machine translation benchmarks have been mostly limited to high-resource or well-represented languages. Despite an increasing interest in low-resource machine translation, there are no standardized reproducible benchmarks for many African languages, many of which are used by millions of speakers but have less digitized textual data. To tackle these challenges, we propose AfroMT, a standardized, clean, and reproducible machine translation benchmark for eight widely spoken African languages. We also develop a suite of analysis tools for system diagnosis taking into account the unique properties of these languages. Furthermore, we explore the newly considered case of low-resource focused pretraining and develop two novel data augmentation-based strategies, leveraging word-level alignment information and pseudo-monolingual data for pretraining multilingual sequence-to-sequence models. We demonstrate significant improvements when pretraining on 11 languages, with gains of up to 2 BLEU points over strong baselines. We also show gains of up to 12 BLEU points over cross-lingual transfer baselines in data-constrained scenarios. All code and pretrained models will be released as further steps towards larger reproducible benchmarks for African languages.