# Token-level Adaptive Training for Neural Machine Translation

arXiv: https://arxiv.org/abs/2010.04380 · HTML: https://mlsanity.com/paper/2010.04380/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2020.

## Built on by (most cited first, 22 of 22)

- [What is Wrong with Perplexity for Long-context Language Modeling?](https://mlsanity.com/paper/2410.23771/) (2024)
- [Provably Robust Multi-bit Watermarking for AI-generated Text](https://mlsanity.com/paper/2401.16820/) (2024)
- [EDT: Improving Large Language Models' Generation by Entropy-based Dynamic Temperature Sampling](https://mlsanity.com/paper/2403.14541/) (2024)
- [Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation](https://mlsanity.com/paper/2106.00903/) (2021)
- [On the Position Bias of On-Policy Distillation](https://mlsanity.com/paper/2606.22600/) (2026)
- [Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translation](https://mlsanity.com/paper/2106.00941/) (2021)
- [Reducing Position Bias in Simultaneous Machine Translation with Length-Aware Framework](https://mlsanity.com/paper/2203.09053/) (2022)
- [Understanding and Bridging the Modality Gap for Speech Translation](https://mlsanity.com/paper/2305.08706/) (2023)
- [An Unsupervised Method for Building Sentence Simplification Corpora in Multiple Languages](https://mlsanity.com/paper/2109.00165/) (2021)
- [Less Learn Shortcut: Analyzing and Mitigating Learning of Spurious Feature-Label Correlation](https://mlsanity.com/paper/2205.12593/) (2022)
- [Frequency-Aware Contrastive Learning for Neural Machine Translation](https://mlsanity.com/paper/2112.14484/) (2021)
- [Mixup Decoding for Diverse Machine Translation](https://mlsanity.com/paper/2109.03402/) (2021)
- [Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training](https://mlsanity.com/paper/2508.15390/) (2025)
- [Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation](https://mlsanity.com/paper/2203.02951/) (2022)
- [Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation](https://mlsanity.com/paper/2105.12523/) (2021)
- [Switching to Discriminative Image Captioning by Relieving a Bottleneck of Reinforcement Learning](https://mlsanity.com/paper/2212.03230/) (2022)
- [Creating a silver standard for patent simplification](https://mlsanity.com/paper/2310.15689/) (2023)
- [Rethinking Personalization in Large Language Models at the Token Level](https://mlsanity.com/paper/2603.06595/) (2026)
- [SFT-GO: Supervised Fine-Tuning with Group Optimization for Large Language Models](https://mlsanity.com/paper/2506.15021/) (2025)
- [Multilingual Lexical Simplification via Paraphrase Generation](https://mlsanity.com/paper/2307.15286/) (2023)
- [Relative Overfitting and Accept-Reject Framework](https://mlsanity.com/paper/2505.07783/) (2025)
- [Sentence Simplification Using Paraphrase Corpus for Initialization](https://mlsanity.com/paper/2305.19754/) (2023)

## Abstract (arXiv)

There exists a token imbalance phenomenon in natural language as different tokens appear with different frequencies, which leads to different learning difficulties for tokens in Neural Machine Translation (NMT). The vanilla NMT model usually adopts trivial equal-weighted objectives for target tokens with different frequencies and tends to generate more high-frequency tokens and less low-frequency tokens compared with the golden token distribution. However, low-frequency tokens may carry critical semantic information that will affect the translation quality once they are neglected. In this paper, we explored target token-level adaptive objectives based on token frequencies to assign appropriate weights for each target token during training. We aimed that those meaningful but relatively low-frequency words could be assigned with larger weights in objectives to encourage the model to pay more attention to these tokens. Our method yields consistent improvements in translation quality on ZH-EN, EN-RO, and EN-DE translation tasks, especially on sentences that contain more low-frequency tokens where we can get 1.68, 1.02, and 0.52 BLEU increases compared with baseline, respectively. Further analyses show that our method can also improve the lexical diversity of translation.