# On the Shortcut Learning in Multilingual Neural Machine Translation

arXiv: https://arxiv.org/abs/2411.10581 · HTML: https://mlsanity.com/paper/2411.10581/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 15 Nov 2024.

## Builds on (most cited first, 29 of 29)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis](https://mlsanity.com/paper/2304.04675/) (2023)
- [Shortcut Learning of Large Language Models in Natural Language Understanding](https://mlsanity.com/paper/2208.11857/) (2022)
- [Look at the First Sentence: Position Bias in Question Answering](https://mlsanity.com/paper/2004.14602/) (2020)
- [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://mlsanity.com/paper/1611.04798/) (2016)
- [Adapting Large Language Models for Document-Level Machine Translation](https://mlsanity.com/paper/2401.06468/) (2024)
- [Why Machine Reading Comprehension Models Learn Shortcuts?](https://mlsanity.com/paper/2106.01024/) (2021)
- [Improving Zero-Shot Translation by Disentangling Positional Information](https://mlsanity.com/paper/2012.15127/) (2020)
- [Improving Multilingual Translation by Representation and Gradient Regularization](https://mlsanity.com/paper/2109.04778/) (2021)
- [Language Tags Matter for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2106.07930/) (2021)
- [Spurious Correlations in Reference-Free Evaluation of Text Generation](https://mlsanity.com/paper/2204.09890/) (2022)
- [On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations](https://mlsanity.com/paper/2204.12708/) (2022)
- [Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation](https://mlsanity.com/paper/2203.03910/) (2022)
- [On the Copying Behaviors of Pre-Training for Neural Machine Translation](https://mlsanity.com/paper/2107.08212/) (2021)
- [Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variables](https://mlsanity.com/paper/2109.04705/) (2021)
- [Understanding and Improving Sequence-to-Sequence Pretraining for Neural Machine Translation](https://mlsanity.com/paper/2203.08442/) (2022)
- [Understanding and Mitigating the Uncertainty in Zero-Shot Translation](https://mlsanity.com/paper/2205.10068/) (2022)

## Abstract (arXiv)

In this study, we revisit the commonly-cited off-target issue in multilingual neural machine translation (MNMT). By carefully designing experiments on different MNMT scenarios and models, we attribute the off-target issue to the overfitting of the shortcuts of (non-centric, centric) language mappings. Specifically, the learned shortcuts biases MNMT to mistakenly translate non-centric languages into the centric language instead of the expected non-centric language for zero-shot translation. Analyses on learning dynamics show that the shortcut learning generally occurs in the later stage of model training, and multilingual pretraining accelerates and aggravates the shortcut learning. Based on these observations, we propose a simple and effective training strategy to eliminate the shortcuts in MNMT models by leveraging the forgetting nature of model training. The only difference from the standard training is that we remove the training instances that may induce the shortcut learning in the later stage of model training. Without introducing any additional data and computational costs, our approach can consistently and significantly improve the zero-shot translation performance by alleviating the shortcut learning for different MNMT models and benchmarks.