# Token Drop mechanism for Neural Machine Translation

arXiv: https://arxiv.org/abs/2010.11018 · HTML: https://mlsanity.com/paper/2010.11018/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2020.

## Built on by (most cited first, 8 of 8)

- [CipherDAug: Ciphertext based Data Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2204.00665/) (2022)
- [Rethinking Data Augmentation for Low-Resource Neural Machine Translation: A Multi-Task Learning Approach](https://mlsanity.com/paper/2109.03645/) (2021)
- [A Survey on Dropout Methods and Experimental Verification in Recommendation](https://mlsanity.com/paper/2204.02027/) (2022)
- [Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws](https://mlsanity.com/paper/2606.06888/) (2026)
- [Joint Dropout: Improving Generalizability in Low-Resource Neural Machine Translation through Phrase Pair Variables](https://mlsanity.com/paper/2307.12835/) (2023)
- [Non-Fluent Synthetic Target-Language Data Improve Neural Machine Translation](https://mlsanity.com/paper/2401.16086/) (2024)
- [Training Language Models with homotokens Leads to Delayed Overfitting](https://mlsanity.com/paper/2601.02867/) (2026)
- [Leveraging Sentence-oriented Augmentation and Transformer-Based Architecture for Vietnamese-Bahnaric Translation](https://mlsanity.com/paper/2601.19124/) (2026)

## Abstract (arXiv)

Neural machine translation with millions of parameters is vulnerable to unfamiliar inputs. We propose Token Drop to improve generalization and avoid overfitting for the NMT model. Similar to word dropout, whereas we replace dropped token with a special token instead of setting zero to words. We further introduce two self-supervised objectives: Replaced Token Detection and Dropped Token Prediction. Our method aims to force model generating target translation with less information, in this way the model can learn textual representation better. Experiments on Chinese-English and English-Romanian benchmark demonstrate the effectiveness of our approach and our model achieves significant improvements over a strong Transformer baseline.