# Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation

arXiv: https://arxiv.org/abs/2105.12523 · HTML: https://mlsanity.com/paper/2105.12523/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 26 May 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Bridging the Gap between Training and Inference for Neural Machine Translation](https://mlsanity.com/paper/1906.02448/) (2019)
- [The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation](https://mlsanity.com/paper/1804.09849/) (2018)
- [Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation](https://mlsanity.com/paper/1906.12068/) (2019)
- [Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss](https://mlsanity.com/paper/1902.09191/) (2019)
- [Self-Paced Learning for Neural Machine Translation](https://mlsanity.com/paper/2010.04505/) (2020)
- [Token-level Adaptive Training for Neural Machine Translation](https://mlsanity.com/paper/2010.04380/) (2020)
- [Towards Understanding Neural Machine Translation with Word Importance](https://mlsanity.com/paper/1909.00326/) (2019)
- [DTMT: A Novel Deep Transition Architecture for Neural Machine Translation](https://mlsanity.com/paper/1812.07807/) (2018)
- [Multi-Unit Transformers for Neural Machine Translation](https://mlsanity.com/paper/2010.10743/) (2020)

## Built on by (most cited first, 5 of 5)

- [Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning](https://mlsanity.com/paper/2211.15076/) (2022)
- [Understanding and Bridging the Modality Gap for Speech Translation](https://mlsanity.com/paper/2305.08706/) (2023)
- [Frequency-Aware Contrastive Learning for Neural Machine Translation](https://mlsanity.com/paper/2112.14484/) (2021)
- [Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation](https://mlsanity.com/paper/2203.02951/) (2022)
- [Specializing Multi-domain NMT via Penalizing Low Mutual Information](https://mlsanity.com/paper/2210.12910/) (2022)

## Abstract (arXiv)

Recently, token-level adaptive training has achieved promising improvement in machine translation, where the cross-entropy loss function is adjusted by assigning different training weights to different tokens, in order to alleviate the token imbalance problem. However, previous approaches only use static word frequency information in the target language without considering the source language, which is insufficient for bilingual tasks like machine translation. In this paper, we propose a novel bilingual mutual information (BMI) based adaptive objective, which measures the learning difficulty for each target token from the perspective of bilingualism, and assigns an adaptive weight accordingly to improve token-level adaptive training. This method assigns larger training weights to tokens with higher BMI, so that easy tokens are updated with coarse granularity while difficult tokens are updated with fine granularity. Experimental results on WMT14 English-to-German and WMT19 Chinese-to-English demonstrate the superiority of our approach compared with the Transformer baseline and previous token-level adaptive training approaches. Further analyses confirm that our method can improve the lexical diversity.