# Reinforced Curriculum Learning on Pre-trained Neural Machine Translation Models

arXiv: https://arxiv.org/abs/2004.05757 · HTML: https://mlsanity.com/paper/2004.05757/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2020.

## Built on by (most cited first, 20 of 20)

- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Efficient Methods for Natural Language Processing: A Survey](https://mlsanity.com/paper/2209.00099/) (2022)
- [Competence-based Multimodal Curriculum Learning for Medical Report Generation](https://mlsanity.com/paper/2206.14579/) (2022)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [A Survey of Active Learning for Natural Language Processing](https://mlsanity.com/paper/2210.10109/) (2022)
- [The Efficiency Spectrum of Large Language Models: An Algorithmic Survey](https://mlsanity.com/paper/2312.00678/) (2023)
- [Boost Your Human Image Generation Model via Direct Preference Optimization](https://mlsanity.com/paper/2405.20216/) (2024)
- [Data Selection Curriculum for Neural Machine Translation](https://mlsanity.com/paper/2203.13867/) (2022)
- [Reinforcement Learning for Generative AI: A Survey](https://mlsanity.com/paper/2308.14328/) (2023)
- [Improving Automated Program Repair with Domain Adaptation](https://mlsanity.com/paper/2212.11414/) (2022)
- [SteloCoder: a Decoder-Only LLM for Multi-Language to Python Code Translation](https://mlsanity.com/paper/2310.15539/) (2023)
- [Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models](https://mlsanity.com/paper/2305.12544/) (2023)
- [A Psychology-based Unified Dynamic Framework for Curriculum Learning](https://mlsanity.com/paper/2408.05326/) (2024)
- [DiscrimLoss: A Universal Loss for Hard Samples and Incorrect Samples Discrimination](https://mlsanity.com/paper/2208.09884/) (2022)
- [Learning to Selectively Learn for Weakly-supervised Paraphrase Generation](https://mlsanity.com/paper/2109.12457/) (2021)
- [Scheduled Curiosity-Deep Dyna-Q: Efficient Exploration for Dialog Policy Learning](https://mlsanity.com/paper/2402.00085/) (2024)
- [Progressively Efficient Learning](https://mlsanity.com/paper/2310.13004/) (2023)
- [False Positive Sampling-based Data Augmentation for Enhanced 3D Object Detection Accuracy](https://mlsanity.com/paper/2403.02639/) (2024)
- [Optimizing the Training Schedule of Multilingual NMT using Reinforcement Learning](https://mlsanity.com/paper/2410.06118/) (2024)
- [Learning What to Fail On: Failure-Mode Contextual Bandits for Adversarial Data Curation](https://mlsanity.com/paper/2608.18681/) (2026)

## Abstract (arXiv)

The competitive performance of neural machine translation (NMT) critically relies on large amounts of training data. However, acquiring high-quality translation pairs requires expert knowledge and is costly. Therefore, how to best utilize a given dataset of samples with diverse quality and characteristics becomes an important yet understudied question in NMT. Curriculum learning methods have been introduced to NMT to optimize a model's performance by prescribing the data input order, based on heuristics such as the assessment of noise and difficulty levels. However, existing methods require training from scratch, while in practice most NMT models are pre-trained on big data already. Moreover, as heuristics, they do not generalize well. In this paper, we aim to learn a curriculum for improving a pre-trained NMT model by re-selecting influential data samples from the original training set and formulate this task as a reinforcement learning problem. Specifically, we propose a data selection framework based on Deterministic Actor-Critic, in which a critic network predicts the expected change of model performance due to a certain sample, while an actor network learns to select the best sample out of a random batch of samples presented to it. Experiments on several translation datasets show that our method can further improve the performance of NMT when original batch training reaches its ceiling, without using additional new training data, and significantly outperforms several strong baseline methods.