# Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translation

arXiv: https://arxiv.org/abs/2010.02552 · HTML: https://mlsanity.com/paper/2010.02552/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 17 of 17)

- [Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation](https://mlsanity.com/paper/2106.00903/) (2021)
- [Aligning Neural Machine Translation Models: Human Feedback in Training and Inference](https://mlsanity.com/paper/2311.09132/) (2023)
- [TranSmart: A Practical Interactive Machine Translation System](https://mlsanity.com/paper/2105.13072/) (2021)
- [Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translation](https://mlsanity.com/paper/2106.00941/) (2021)
- [Cross-modality Data Augmentation for End-to-End Sign Language Translation](https://mlsanity.com/paper/2305.11096/) (2023)
- [On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translation](https://mlsanity.com/paper/2209.03316/) (2022)
- [Understanding and Improving Sequence-to-Sequence Pretraining for Neural Machine Translation](https://mlsanity.com/paper/2203.08442/) (2022)
- [Tencent's Multilingual Machine Translation System for WMT22 Large-Scale African Languages](https://mlsanity.com/paper/2210.09644/) (2022)
- [Data Selection Curriculum for Neural Machine Translation](https://mlsanity.com/paper/2203.13867/) (2022)
- [Confidence-Aware Scheduled Sampling for Neural Machine Translation](https://mlsanity.com/paper/2107.10427/) (2021)
- [Understanding and Mitigating the Uncertainty in Zero-Shot Translation](https://mlsanity.com/paper/2205.10068/) (2022)
- [Tencent AI Lab - Shanghai Jiao Tong University Low-Resource Translation System for the WMT22 Translation Task](https://mlsanity.com/paper/2210.08742/) (2022)
- [Scaling Back-Translation with Domain Text Generation for Sign Language Gloss Translation](https://mlsanity.com/paper/2210.07054/) (2022)
- [BitextEdit: Automatic Bitext Editing for Improved Low-Resource Machine Translation](https://mlsanity.com/paper/2111.06787/) (2021)
- [Can Synthetic Translations Improve Bitext Quality?](https://mlsanity.com/paper/2203.07643/) (2022)
- [STAD: Self-Training with Ambiguous Data for Low-Resource Relation Extraction](https://mlsanity.com/paper/2209.01431/) (2022)
- [Towards Reliable Neural Machine Translation with Consistency-Aware Meta-Learning](https://mlsanity.com/paper/2303.10966/) (2023)

## Abstract (arXiv)

Large-scale training datasets lie at the core of the recent success of neural machine translation (NMT) models. However, the complex patterns and potential noises in the large-scale data make training NMT models difficult. In this work, we explore to identify the inactive training examples which contribute less to the model performance, and show that the existence of inactive examples depends on the data distribution. We further introduce data rejuvenation to improve the training of NMT models on large-scale datasets by exploiting inactive examples. The proposed framework consists of three phases. First, we train an identification model on the original training data, and use it to distinguish inactive examples and active examples by their sentence-level output probabilities. Then, we train a rejuvenation model on the active examples, which is used to re-label the inactive examples with forward-translation. Finally, the rejuvenated examples and the active examples are combined to train the final NMT model. Experimental results on WMT14 English-German and English-French datasets show that the proposed data rejuvenation consistently and significantly improves performance for several strong NMT models. Extensive analyses reveal that our approach stabilizes and accelerates the training process of NMT models, resulting in final models with better generalization capability.