# HLT-MT: High-resource Language-specific Training for Multilingual Neural Machine Translation

arXiv: https://arxiv.org/abs/2207.04906 · HTML: https://mlsanity.com/paper/2207.04906/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 11 Jul 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism](https://mlsanity.com/paper/1601.01073/) (2016)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information](https://mlsanity.com/paper/2010.03142/) (2020)
- [Multilingual Neural Machine Translation with Language Clustering](https://mlsanity.com/paper/1908.09324/) (2019)
- [Multi-task Learning for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2010.02523/) (2020)
- [XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders](https://mlsanity.com/paper/2012.15547/) (2020)
- [Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoders](https://mlsanity.com/paper/2004.06575/) (2020)
- [Multilingual NMT with a language-independent attention bridge](https://mlsanity.com/paper/1811.00498/) (2018)
- [Multilingual Neural Machine Translation with Deep Encoder and Multiple Shallow Decoders](https://mlsanity.com/paper/2206.02079/) (2022)
- [Adaptive Sparse Transformer for Multilingual Translation](https://mlsanity.com/paper/2104.07358/) (2021)

## Built on by (most cited first, 4 of 4)

- [GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator](https://mlsanity.com/paper/2212.10218/) (2022)
- [Adaptive Neural Ranking Framework: Toward Maximized Business Goal for Cascade Ranking Systems](https://mlsanity.com/paper/2310.10462/) (2023)
- [HanoiT: Enhancing Context-aware Translation via Selective Context](https://mlsanity.com/paper/2301.06825/) (2023)
- [University of Cape Town's WMT22 System: Multilingual Machine Translation for Southern African Languages](https://mlsanity.com/paper/2210.11757/) (2022)

## Abstract (arXiv)

Multilingual neural machine translation (MNMT) trained in multiple language pairs has attracted considerable attention due to fewer model parameters and lower training costs by sharing knowledge among multiple languages. Nonetheless, multilingual training is plagued by language interference degeneration in shared parameters because of the negative interference among different translation directions, especially on high-resource languages. In this paper, we propose the multilingual translation model with the high-resource language-specific training (HLT-MT) to alleviate the negative interference, which adopts the two-stage training with the language-specific selection mechanism. Specifically, we first train the multilingual model only with the high-resource pairs and select the language-specific modules at the top of the decoder to enhance the translation quality of high-resource directions. Next, the model is further trained on all available corpora to transfer knowledge from high-resource languages (HRLs) to low-resource languages (LRLs). Experimental results show that HLT-MT outperforms various strong baselines on WMT-10 and OPUS-100 benchmarks. Furthermore, the analytic experiments validate the effectiveness of our method in mitigating the negative interference in multilingual training.