# Focus on the Target's Vocabulary: Masked Label Smoothing for Machine Translation

arXiv: https://arxiv.org/abs/2203.02889 · HTML: https://mlsanity.com/paper/2203.02889/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 6 Mar 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [Reward Augmented Maximum Likelihood for Neural Structured Prediction](https://mlsanity.com/paper/1609.00150/) (2016)
- [Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information](https://mlsanity.com/paper/2010.03142/) (2020)
- [On the Inference Calibration of Neural Machine Translation](https://mlsanity.com/paper/2005.00963/) (2020)
- [Classical Structured Prediction Losses for Sequence to Sequence Learning](https://mlsanity.com/paper/1711.04956/) (2017)
- [DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders](https://mlsanity.com/paper/2106.13736/) (2021)
- [Generalized Entropy Regularization or: There's Nothing Special about Label Smoothing](https://mlsanity.com/paper/2005.00820/) (2020)
- [Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies](https://mlsanity.com/paper/1905.05475/) (2019)
- [Semantic Label Smoothing for Sequence to Sequence Problems](https://mlsanity.com/paper/2010.07447/) (2020)
- [Shared-Private Bilingual Word Embeddings for Neural Machine Translation](https://mlsanity.com/paper/1906.03100/) (2019)

## Built on by (most cited first, 3 of 3)

- [Understanding and Mitigating the Uncertainty in Zero-Shot Translation](https://mlsanity.com/paper/2205.10068/) (2022)
- [On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation](https://mlsanity.com/paper/2305.10930/) (2023)
- [LABO: Towards Learning Optimal Label Regularization via Bi-level Optimization](https://mlsanity.com/paper/2305.04971/) (2023)

## Abstract (arXiv)

Label smoothing and vocabulary sharing are two widely used techniques in neural machine translation models. However, we argue that simply applying both techniques can be conflicting and even leads to sub-optimal performance. When allocating smoothed probability, original label smoothing treats the source-side words that would never appear in the target language equally to the real target-side words, which could bias the translation model. To address this issue, we propose Masked Label Smoothing (MLS), a new mechanism that masks the soft label probability of source-side words to zero. Simple yet effective, MLS manages to better integrate label smoothing with vocabulary sharing. Our extensive experiments show that MLS consistently yields improvement over original label smoothing on different datasets, including bilingual and multilingual translation from both translation quality and model's calibration. Our code is released at https://github.com/PKUnlp-icler/MLS