# Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?

arXiv: https://arxiv.org/abs/2203.08850 · HTML: https://mlsanity.com/paper/2203.08850/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 47 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://mlsanity.com/paper/1911.06154/) (2019)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Rapid Adaptation of Neural Machine Translation to New Languages](https://mlsanity.com/paper/1808.04189/) (2018)
- [On the Impact of Various Types of Noise on Neural Machine Translation](https://mlsanity.com/paper/1805.12282/) (2018)
- [Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation](https://mlsanity.com/paper/2010.10907/) (2020)
- [Investigating Multilingual NMT Representations at Scale](https://mlsanity.com/paper/1909.02197/) (2019)
- [IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation](https://mlsanity.com/paper/2104.08200/) (2021)
- [How to Adapt Your Pretrained Multilingual Model to 1600 Languages](https://mlsanity.com/paper/2106.02124/) (2021)
- [PMIndia -- A Collection of Parallel Corpora of Languages of India](https://mlsanity.com/paper/2001.09907/) (2020)
- [The Effect of Domain and Diacritics in Yor`ub'a-English Neural Machine Translation](https://mlsanity.com/paper/2103.08647/) (2021)
- [Filtering and Mining Parallel Data in a Joint Multilingual Space](https://mlsanity.com/paper/1805.09822/) (2018)
- [Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2105.03953/) (2021)
- [Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set](https://mlsanity.com/paper/1909.01522/) (2019)
- [Data Augmentation and Terminology Integration for Domain-Specific Sinhala-English-Tamil Statistical Machine Translation](https://mlsanity.com/paper/2011.02821/) (2020)
- [Benchmarking Neural Machine Translation for Southern African Languages](https://mlsanity.com/paper/1906.10511/) (2019)
- [Evaluating the Supervised and Zero-shot Performance of Multi-lingual Translation Models](https://mlsanity.com/paper/1906.09675/) (2019)
- [Family of Origin and Family of Choice: Massively Parallel Lexiconized Iterative Pretraining for Severely Low Resource Machine Translation](https://mlsanity.com/paper/2104.05848/) (2021)

## Built on by (most cited first, 24 of 24)

- [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://mlsanity.com/paper/2309.07445/) (2023)
- [A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation](https://mlsanity.com/paper/2205.02022/) (2022)
- [IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models](https://mlsanity.com/paper/2406.03368/) (2024)
- [Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMs](https://mlsanity.com/paper/2311.03127/) (2023)
- [From One to Many: Expanding the Scope of Toxicity Mitigation in Language Models](https://mlsanity.com/paper/2403.03893/) (2024)
- [Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora](https://mlsanity.com/paper/2402.07446/) (2024)
- [Towards a Better Understanding of Variations in Zero-Shot Neural Machine Translation Performance](https://mlsanity.com/paper/2310.10385/) (2023)
- [Sinhala Transliteration: A Comparative Analysis Between Rule-based and Seq2Seq Approaches](https://mlsanity.com/paper/2501.00529/) (2024)
- [Leveraging Auxiliary Domain Parallel Data in Intermediate Task Fine-tuning for Low-resource Translation](https://mlsanity.com/paper/2306.01382/) (2023)
- [Does Generative AI speak Nigerian-Pidgin?: Issues about Representativeness and Bias for Multilingualism in LLMs](https://mlsanity.com/paper/2404.19442/) (2024)
- [Meeting the Needs of Low-Resource Languages: The Value of Automatic Alignments via Pretrained Models](https://mlsanity.com/paper/2302.07912/) (2023)
- [Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation](https://mlsanity.com/paper/2404.04212/) (2024)
- [How to Learn in a Noisy World? Self-Correcting the Real-World Data Noise in Machine Translation](https://mlsanity.com/paper/2407.02208/) (2024)
- [Specification-Aware Machine Translation and Evaluation for Purpose Alignment](https://mlsanity.com/paper/2509.17559/) (2025)
- [SiTSE: Sinhala Text Simplification Dataset and Evaluation](https://mlsanity.com/paper/2412.01293/) (2024)
- [Language Fusion for Parameter-Efficient Cross-lingual Transfer](https://mlsanity.com/paper/2501.06892/) (2025)
- [Exploiting Domain-Specific Parallel Data on Multilingual Language Models for Low-resource Language Translation](https://mlsanity.com/paper/2412.19522/) (2024)
- [Beyond Vanilla Fine-Tuning: Leveraging Multistage, Multilingual, and Domain-Specific Methods for Low-Resource Machine Translation](https://mlsanity.com/paper/2503.22582/) (2025)
- [Zero-Shot Performance Prediction for Probabilistic Scaling Laws](https://mlsanity.com/paper/2510.16743/) (2025)
- [Harnessing the Power of Artificial Intelligence to Vitalize Endangered Indigenous Languages: Technologies and Experiences](https://mlsanity.com/paper/2407.12620/) (2024)
- [ChatZero:Zero-shot Cross-Lingual Dialogue Generation via Pseudo-Target Language](https://mlsanity.com/paper/2408.08724/) (2024)
- [A Data Selection Approach for Enhancing Low Resource Machine Translation Using Cross-Lingual Sentence Representations](https://mlsanity.com/paper/2409.02712/) (2024)
- [SiniticMTError: A Machine Translation Dataset with Error Annotations for Sinitic Languages](https://mlsanity.com/paper/2509.20557/) (2025)
- [EnSiTa - A Trilingual Multi-Domain Parallel Dataset and Benchmark for Domain-Specific Machine Translation](https://mlsanity.com/paper/2609.29511/) (2026)

## Abstract (arXiv)

What can pre-trained multilingual sequence-to-sequence models like mBART contribute to translating low-resource languages? We conduct a thorough empirical experiment in 10 languages to ascertain this, considering five factors: (1) the amount of fine-tuning data, (2) the noise in the fine-tuning data, (3) the amount of pre-training data in the model, (4) the impact of domain mismatch, and (5) language typology. In addition to yielding several heuristics, the experiments form a framework for evaluating the data sensitivities of machine translation systems. While mBART is robust to domain differences, its translations for unseen and typologically distant languages remain below 3.0 BLEU. In answer to our title's question, mBART is not a low-resource panacea; we therefore encourage shifting the emphasis from new models to new data.