Why Neural Machine Translation Prefers Empty Outputs
Not on the map. Why?
Abstract (arXiv)
We investigate why neural machine translation (NMT) systems assign high probability to empty translations. We find two explanations. First, label smoothing makes correct-length translations less confident, making it easier for the empty translation to finally outscore them. Second, NMT systems use the same, high-frequency EoS word to end all target sentences, regardless of length. This creates an implicit smoothing that increases zero-length translations. Using different EoS types in target sentences of different lengths exposes and eliminates this implicit smoothing.
Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Dec 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 4; 2022: 1; 2023: 1.
6 papers.
- 2021Why don't people use character-level machine translation?cited by 27
- 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19
- 2021Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modelingcited by 10
- 2021What Do You Get When You Cross Beam Search with Nucleus Sampling?cited by 7
- 2022The Implicit Length Bias of Label Smoothing on Beam Search Decodingcited by 2
- 2021Mode recovery in neural autoregressive sequence modelingcited by 1
- 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?31 Jul 2023
- 2022The Implicit Length Bias of Label Smoothing on Beam Search Decoding2 May 2022
- 2021Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling16 Dec 2021
- 2021Why don't people use character-level machine translation?15 Oct 2021
- 2021What Do You Get When You Cross Beam Search with Nucleus Sampling?20 Jul 2021
- 2021Mode recovery in neural autoregressive sequence modeling10 Jun 2021