# Exploiting Domain-Specific Parallel Data on Multilingual Language Models for Low-resource Language Translation

arXiv: https://arxiv.org/abs/2412.19522 · HTML: https://mlsanity.com/paper/2412.19522/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 50 references that are in the map.
- First posted on arXiv 27 Dec 2024.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks](https://mlsanity.com/paper/1811.01088/) (2018)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?](https://mlsanity.com/paper/2005.00628/) (2020)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation](https://mlsanity.com/paper/2205.02022/) (2022)
- [Survey of Low-Resource Machine Translation](https://mlsanity.com/paper/2109.00486/) (2021)
- [From LLM to NMT: Advancing Low-Resource Machine Translation with Claude](https://mlsanity.com/paper/2404.13813/) (2024)
- [Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP World](https://mlsanity.com/paper/2210.08523/) (2022)
- [PMIndia -- A Collection of Parallel Corpora of Languages of India](https://mlsanity.com/paper/2001.09907/) (2020)
- [MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs](https://mlsanity.com/paper/2104.08692/) (2021)
- [Language Model Prior for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2004.14928/) (2020)
- [Semi-Supervised Learning for Neural Machine Translation](https://mlsanity.com/paper/1606.04596/) (2016)
- [Neural Machine Translation with Pivot Languages](https://mlsanity.com/paper/1611.04928/) (2016)
- [Domain Divergences: a Survey and Empirical Analysis](https://mlsanity.com/paper/2010.12198/) (2020)
- [Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?](https://mlsanity.com/paper/2203.08850/) (2022)
- [nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?](https://mlsanity.com/paper/2106.02171/) (2021)
- [PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining](https://mlsanity.com/paper/2108.01887/) (2021)
- [Data Augmentation and Terminology Integration for Domain-Specific Sinhala-English-Tamil Statistical Machine Translation](https://mlsanity.com/paper/2011.02821/) (2020)
- [Predicting Machine Translation Performance on Low-Resource Languages: The Role of Domain Similarity](https://mlsanity.com/paper/2402.02633/) (2024)
- [Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora](https://mlsanity.com/paper/2402.07446/) (2024)
- [Exploiting Out-of-Domain Parallel Data through Multilingual Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1907.03060/) (2019)
- [Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking](https://mlsanity.com/paper/2109.13620/) (2021)

## Built on by (most cited first, 2 of 2)

- [EnSiTa - A Trilingual Multi-Domain Parallel Dataset and Benchmark for Domain-Specific Machine Translation](https://mlsanity.com/paper/2609.29511/) (2026)
- [Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs](https://mlsanity.com/paper/2605.24681/) (2026)

## Abstract (arXiv)

Neural Machine Translation (NMT) systems built on multilingual sequence-to-sequence Language Models (msLMs) fail to deliver expected results when the amount of parallel data for a language, as well as the language's representation in the model are limited. This restricts the capabilities of domain-specific NMT systems for low-resource languages (LRLs). As a solution, parallel data from auxiliary domains can be used either to fine-tune or to further pre-train the msLM. We present an evaluation of the effectiveness of these two techniques in the context of domain-specific LRL-NMT. We also explore the impact of domain divergence on NMT model performance. We recommend several strategies for utilizing auxiliary parallel data in building domain-specific NMT models for LRLs.