The Unreasonable Volatility of Neural Machine Translation Models
Not on the map. Why?
Abstract (arXiv)
Recent works have shown that Neural Machine Translation (NMT) models achieve impressive performance, however, questions about understanding the behavior of these models remain unanswered. We investigate the unexpected volatility of NMT models where the input is semantically and syntactically correct. We discover that with trivial modifications of source sentences, we can identify cases where textitunexpected changes happen in the translation and in the worst case lead to mistranslations. This volatile behavior of translating extremely similar sentences in surprisingly different ways highlights the underlying generalization problem of current NMT models. We find that both RNN and Transformer models display volatile behavior in 26% and 19% of sentence variations, respectively.
Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 May 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 4; 2022: 3; 2023: 1; 2024: 1.
9 papers.
- 2021The paradox of the compositionality of natural language: a neural machine translation case studycited by 58
- 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
- 2021On Compositional Generalization of Neural Machine Translationcited by 34
- 2022How sensitive are translation systems to extra contexts? Mitigating gender bias in Neural Machine Translation models through relevant contextscited by 10
- 2022Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspectivecited by 5
- 2024Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMscited by 5
- 2021Sometimes We Want Translationesecited by 3
- 2022Categorizing Semantic Representations for Neural Machine Translationcited by 2
- 2023Joint Dropout: Improving Generalizability in Low-Resource Neural Machine Translation through Phrase Pair Variablescited by 1
- 2024Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMs24 Jul 2024
- 2023Joint Dropout: Improving Generalizability in Low-Resource Neural Machine Translation through Phrase Pair Variables24 Jul 2023
- 2022Categorizing Semantic Representations for Neural Machine Translation13 Oct 2022
- 2022How sensitive are translation systems to extra contexts? Mitigating gender bias in Neural Machine Translation models through relevant contexts22 May 2022
- 2022Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspective26 Apr 2022
- 2021The paradox of the compositionality of natural language: a neural machine translation case study12 Aug 2021
- 2021On Compositional Generalization of Neural Machine Translation31 May 2021
- 2021Sometimes We Want Translationese15 Apr 2021
- 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey14 Apr 2021