mlsanityData to 7 Oct 2026

The Implicit Length Bias of Label Smoothing on Beam Search Decoding

arXiv 2205.00659 ↗First posted 2 May 2022cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Label smoothing is ubiquitously applied in Neural Machine Translation (NMT) training. While label smoothing offers a desired regularization effect during model training, in this paper we demonstrate that it nevertheless introduces length biases in the beam search decoding procedure. Our analysis shows that label smoothing implicitly applies a length penalty term to output sequence, causing a bias towards shorter translations. We also show that for a model fully optimized with label smoothing, translation length is implicitly upper bounded by a fixed constant independent of input. We verify our theory by applying a simple rectification function at inference time to restore the unbiased distributions from the label-smoothed model predictions. This rectification method led to consistent quality improvements on WMT English-German, English-French, English-Czech and English-Chinese tasks, up to +0.3 BLEU at beam size 4 and +2.8 BLEU at beam size 200.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 23 of its 28 references that are in the map. First posted on arXiv 2 May 2022.

Builds on

23 of 28 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  6. 2019When Does Label Smoothing Help?cited by 985
  7. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  8. 2017Six Challenges for Neural Machine Translationcited by 334
  9. 2020Does label smoothing mitigate label noise?cited by 204
  10. 2018Analyzing Uncertainty in Neural Machine Translationcited by 126
  11. 2018Correcting Length Bias in Neural Machine Translationcited by 119
  12. 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
11 more
  1. 2020If beam search is the answer, what was the question?cited by 114
  2. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 101
  3. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  4. 2020On the Inference Calibration of Neural Machine Translationcited by 61
  5. 2020Generalized Entropy Regularization or: There's Nothing Special about Label Smoothingcited by 40
  6. 2018Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translationcited by 29
  7. 2020Semantic Label Smoothing for Sequence to Sequence Problemscited by 13
  8. 2020An Investigation of how Label Smoothing Affects Generalizationcited by 11
  9. 2021Smoothing and Shrinking the Sparse Seq2Seq Search Spacecited by 11
  10. 2021Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modelingcited by 10
  11. 2020Why Neural Machine Translation Prefers Empty Outputscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducerscited by 1
  2. 2026LBR: Towards Mitigating Length Bias in Large Language Models for Recommendationcited by 0