# A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation

arXiv: https://arxiv.org/abs/2205.02022 · HTML: https://mlsanity.com/paper/2205.02022/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 76 references that are in the map.
- First posted on arXiv 4 May 2022.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages](https://mlsanity.com/paper/2010.02353/) (2020)
- [MasakhaNER: Named Entity Recognition for African Languages](https://mlsanity.com/paper/2103.11811/) (2021)
- [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://mlsanity.com/paper/1911.06154/) (2019)
- [Achieving Human Parity on Automatic Chinese to English News Translation](https://mlsanity.com/paper/1803.05567/) (2018)
- [Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism](https://mlsanity.com/paper/1601.01073/) (2016)
- [Survey of Low-Resource Machine Translation](https://mlsanity.com/paper/2109.00486/) (2021)
- [Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation](https://mlsanity.com/paper/1808.07048/) (2018)
- [Extending Multilingual BERT to Low-Resource Languages](https://mlsanity.com/paper/2004.13640/) (2020)
- [TICO-19: the Translation Initiative for Covid-19](https://mlsanity.com/paper/2007.01788/) (2020)
- [Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation](https://mlsanity.com/paper/1808.10432/) (2018)
- [IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation](https://mlsanity.com/paper/2104.08200/) (2021)
- [Facebook AI WMT21 News Translation Task Submission](https://mlsanity.com/paper/2108.03265/) (2021)
- [Joey NMT: A Minimalist NMT Toolkit for Novices](https://mlsanity.com/paper/1907.12484/) (2019)

## Built on by (most cited first, 30 of 98)

- [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://mlsanity.com/paper/2309.07445/) (2023)
- [Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages](https://mlsanity.com/paper/2305.12182/) (2023)
- [LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimization](https://mlsanity.com/paper/2306.01102/) (2023)
- [Building Machine Translation Systems for the Next Thousand Languages](https://mlsanity.com/paper/2205.03983/) (2022)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages](https://mlsanity.com/paper/2211.03263/) (2022)
- [AfroBench: How Good are Large Language Models on African Languages?](https://mlsanity.com/paper/2311.07978/) (2023)
- [IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models](https://mlsanity.com/paper/2406.03368/) (2024)
- [SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)](https://mlsanity.com/paper/2304.06845/) (2023)
- [MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition](https://mlsanity.com/paper/2210.12391/) (2022)
- [MasakhaNEWS: News Topic Classification for African languages](https://mlsanity.com/paper/2304.09972/) (2023)
- [Teaching Large Language Models an Unseen Language on the Fly](https://mlsanity.com/paper/2402.19167/) (2024)
- [Goldfish: Monolingual Language Models for 350 Languages](https://mlsanity.com/paper/2408.10441/) (2024)
- [AfriQA: Cross-lingual Open-Retrieval Question Answering for African Languages](https://mlsanity.com/paper/2305.06897/) (2023)
- [MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African Languages](https://mlsanity.com/paper/2305.13989/) (2023)
- [A Material Lens on Coloniality in NLP](https://mlsanity.com/paper/2311.08391/) (2023)
- [JWSign: A Highly Multilingual Corpus of Bible Translations for more Diversity in Sign Language Processing](https://mlsanity.com/paper/2311.10174/) (2023)
- [InkubaLM: A small language model for low-resource African languages](https://mlsanity.com/paper/2408.17024/) (2024)
- [EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models](https://mlsanity.com/paper/2409.17892/) (2024)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [NollySenti: Leveraging Transfer Learning and Machine Translation for Nigerian Movie Sentiment Classification](https://mlsanity.com/paper/2305.10971/) (2023)
- [MDIA: A Benchmark for Multilingual Dialogue Generation in 46 Languages](https://mlsanity.com/paper/2208.13078/) (2022)
- [Correcting FLORES Evaluation Dataset for Four African Languages](https://mlsanity.com/paper/2409.00626/) (2024)
- [Preparing the Vuk'uzenzele and ZA-gov-multilingual South African multilingual corpora](https://mlsanity.com/paper/2303.03750/) (2023)
- [HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Language](https://mlsanity.com/paper/2305.17690/) (2023)
- [The Zeno's Paradox of `Low-Resource' Languages](https://mlsanity.com/paper/2410.20817/) (2024)
- [AFRIDOC-MT: Document-level MT Corpus for African Languages](https://mlsanity.com/paper/2501.06374/) (2025)
- [TaTa: A Multilingual Table-to-Text Dataset for African Languages](https://mlsanity.com/paper/2211.00142/) (2022)
- [xPQA: Cross-Lingual Product Question Answering across 12 Languages](https://mlsanity.com/paper/2305.09249/) (2023)
- [Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning](https://mlsanity.com/paper/2405.19462/) (2024)

## Abstract (arXiv)

Recent advances in the pre-training of language models leverage large-scale datasets to create multilingual models. However, low-resource languages are mostly left out in these datasets. This is primarily because many widely spoken languages are not well represented on the web and therefore excluded from the large-scale crawls used to create datasets. Furthermore, downstream users of these models are restricted to the selection of languages originally chosen for pre-training. This work investigates how to optimally leverage existing pre-trained models to create low-resource translation systems for 16 African languages. We focus on two questions: 1) How can pre-trained models be used for languages not included in the initial pre-training? and 2) How can the resulting translation models effectively transfer to new domains? To answer these questions, we create a new African news corpus covering 16 languages, of which eight languages are not part of any existing evaluation dataset. We demonstrate that the most effective strategy for transferring both to additional languages and to additional domains is to fine-tune large pre-trained models on small quantities of high-quality translation data.