# Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation

arXiv: https://arxiv.org/abs/2105.15087 · HTML: https://mlsanity.com/paper/2105.15087/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 53 references that are in the map.
- First posted on arXiv 31 May 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [Analyzing Uncertainty in Neural Machine Translation](https://mlsanity.com/paper/1803.00047/) (2018)
- [When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?](https://mlsanity.com/paper/1804.06323/) (2018)
- [On the Impact of Various Types of Noise on Neural Machine Translation](https://mlsanity.com/paper/1805.12282/) (2018)
- [Calibration of Encoder Decoder Models for Neural Machine Translation](https://mlsanity.com/paper/1903.00802/) (2019)
- [On the Inference Calibration of Neural Machine Translation](https://mlsanity.com/paper/2005.00963/) (2020)
- [Linguistic Input Features Improve Neural Machine Translation](https://mlsanity.com/paper/1606.02892/) (2016)
- [Detecting Fine-Grained Cross-Lingual Semantic Divergences without Supervision by Learning to Rank](https://mlsanity.com/paper/2010.03662/) (2020)
- [The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020](https://mlsanity.com/paper/2008.04885/) (2020)
- [Identifying Semantic Divergences in Parallel Text without Annotations](https://mlsanity.com/paper/1803.11112/) (2018)
- [Novel Applications of Factored Neural Machine Translation](https://mlsanity.com/paper/1910.03912/) (2019)
- [Factored Neural Machine Translation](https://mlsanity.com/paper/1609.04621/) (2016)
- [Neural Machine Translation by Generating Multiple Linguistic Factors](https://mlsanity.com/paper/1712.01821/) (2017)
- [Towards Recognizing Phrase Translation Processes: Experiments on English-French](https://mlsanity.com/paper/1904.12213/) (2019)

## Built on by (most cited first, 9 of 9)

- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection](https://mlsanity.com/paper/2301.07779/) (2023)
- [Cultural Adaptation of Recipes](https://mlsanity.com/paper/2310.17353/) (2023)
- [How to Learn in a Noisy World? Self-Correcting the Real-World Data Noise in Machine Translation](https://mlsanity.com/paper/2407.02208/) (2024)
- [BitextEdit: Automatic Bitext Editing for Improved Low-Resource Machine Translation](https://mlsanity.com/paper/2111.06787/) (2021)
- [Learning from Many Voices: Literary MT Using Multi-Reference Human and Synthetic Data](https://mlsanity.com/paper/2412.18707/) (2024)
- [Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representation](https://mlsanity.com/paper/2210.03018/) (2022)
- [Can Synthetic Translations Improve Bitext Quality?](https://mlsanity.com/paper/2203.07643/) (2022)

## Abstract (arXiv)

While it has been shown that Neural Machine Translation (NMT) is highly sensitive to noisy parallel training samples, prior work treats all types of mismatches between source and target as noise. As a result, it remains unclear how samples that are mostly equivalent but contain a small number of semantically divergent tokens impact NMT training. To close this gap, we analyze the impact of different types of fine-grained semantic divergences on Transformer models. We show that models trained on synthetic divergences output degenerated text more frequently and are less confident in their predictions. Based on these findings, we introduce a divergent-aware NMT framework that uses factors to help NMT recover from the degradation caused by naturally occurring divergences, improving both translation quality and model calibration on EN-FR tasks.