# Semantic Label Smoothing for Sequence to Sequence Problems

arXiv: https://arxiv.org/abs/2010.07447 · HTML: https://mlsanity.com/paper/2010.07447/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2020.

## Built on by (most cited first, 13 of 13)

- [Calibrating Sequence likelihood Improves Conditional Language Generation](https://mlsanity.com/paper/2210.00045/) (2022)
- [Smoothing and Shrinking the Sparse Seq2Seq Search Space](https://mlsanity.com/paper/2103.10291/) (2021)
- [DetIE: Multilingual Open Information Extraction Inspired by Object Detection](https://mlsanity.com/paper/2206.12514/) (2022)
- [Language Modelling via Learning to Rank](https://mlsanity.com/paper/2110.06961/) (2021)
- [Context-Aware Selective Label Smoothing for Calibrating Sequence Recognition Model](https://mlsanity.com/paper/2303.06946/) (2023)
- [Rethinking Label Smoothing on Multi-hop Question Answering](https://mlsanity.com/paper/2212.09512/) (2022)
- [Focus on the Target's Vocabulary: Masked Label Smoothing for Machine Translation](https://mlsanity.com/paper/2203.02889/) (2022)
- [The Implicit Length Bias of Label Smoothing on Beam Search Decoding](https://mlsanity.com/paper/2205.00659/) (2022)
- [Label Smoothed Embedding Hypothesis for Out-of-Distribution Detection](https://mlsanity.com/paper/2102.05131/) (2021)
- [Reprint: a randomized extrapolation based on principal components for data augmentation](https://mlsanity.com/paper/2204.12024/) (2022)
- [Perception and Semantic Aware Regularization for Sequential Confidence Calibration](https://mlsanity.com/paper/2305.19498/) (2023)
- [Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Models](https://mlsanity.com/paper/2603.29123/) (2026)
- [Inference-Time Consensus for Mitigating Hidden Behaviors from LLM Fine-Tuning](https://mlsanity.com/paper/2607.23394/) (2026)

## Abstract (arXiv)

Label smoothing has been shown to be an effective regularization strategy in classification, that prevents overfitting and helps in label de-noising. However, extending such methods directly to seq2seq settings, such as Machine Translation, is challenging: the large target output space of such problems makes it intractable to apply label smoothing over all possible outputs. Most existing approaches for seq2seq settings either do token level smoothing, or smooth over sequences generated by randomly substituting tokens in the target sequence. Unlike these works, in this paper, we propose a technique that smooths over well formed relevant sequences that not only have sufficient n-gram overlap with the target sequence, but are also semantically similar. Our method shows a consistent and significant improvement over the state-of-the-art techniques on different datasets.