# Why Neural Machine Translation Prefers Empty Outputs

arXiv: https://arxiv.org/abs/2012.13454 · HTML: https://mlsanity.com/paper/2012.13454/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Dec 2020.

## Built on by (most cited first, 6 of 6)

- [Why don't people use character-level machine translation?](https://mlsanity.com/paper/2110.08191/) (2021)
- [Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?](https://mlsanity.com/paper/2308.00189/) (2023)
- [Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling](https://mlsanity.com/paper/2112.08914/) (2021)
- [What Do You Get When You Cross Beam Search with Nucleus Sampling?](https://mlsanity.com/paper/2107.09729/) (2021)
- [The Implicit Length Bias of Label Smoothing on Beam Search Decoding](https://mlsanity.com/paper/2205.00659/) (2022)
- [Mode recovery in neural autoregressive sequence modeling](https://mlsanity.com/paper/2106.05459/) (2021)

## Abstract (arXiv)

We investigate why neural machine translation (NMT) systems assign high probability to empty translations. We find two explanations. First, label smoothing makes correct-length translations less confident, making it easier for the empty translation to finally outscore them. Second, NMT systems use the same, high-frequency EoS word to end all target sentences, regardless of length. This creates an implicit smoothing that increases zero-length translations. Using different EoS types in target sentences of different lengths exposes and eliminates this implicit smoothing.