# Data Selection Curriculum for Neural Machine Translation

arXiv: https://arxiv.org/abs/2203.13867 · HTML: https://mlsanity.com/paper/2203.13867/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 25 Mar 2022.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [On The Power of Curriculum Learning in Training Deep Networks](https://mlsanity.com/paper/1904.03626/) (2019)
- [Competence-based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1903.09848/) (2019)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [Achieving Human Parity on Automatic Chinese to English News Translation](https://mlsanity.com/paper/1803.05567/) (2018)
- [Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings](https://mlsanity.com/paper/1811.01136/) (2018)
- [On the Impact of Various Types of Noise on Neural Machine Translation](https://mlsanity.com/paper/1805.12282/) (2018)
- [Curriculum Learning and Minibatch Bucketing in Neural Machine Translation](https://mlsanity.com/paper/1707.09533/) (2017)
- [Norm-Based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/2006.02014/) (2020)
- [Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives](https://mlsanity.com/paper/1905.10847/) (2019)
- [An Empirical Exploration of Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1811.00739/) (2018)
- [Visualizing and Understanding Curriculum Learning for Long Short-Term Memory Networks](https://mlsanity.com/paper/1611.06204/) (2016)
- [Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection](https://mlsanity.com/paper/1809.00068/) (2018)
- [Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corpora](https://mlsanity.com/paper/1809.00197/) (2018)
- [Dynamic Data Selection for Neural Machine Translation](https://mlsanity.com/paper/1708.00712/) (2017)
- [Curriculum learning for language modeling](https://mlsanity.com/paper/2108.02170/) (2021)
- [Reinforcement Learning based Curriculum Optimization for Neural Machine Translation](https://mlsanity.com/paper/1903.00041/) (2019)
- [Fast Domain Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1612.06897/) (2016)
- [Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translation](https://mlsanity.com/paper/2103.02262/) (2021)
- [Self-Paced Learning for Neural Machine Translation](https://mlsanity.com/paper/2010.04505/) (2020)
- [Low-Resource Corpus Filtering using Multilingual Sentence Embeddings](https://mlsanity.com/paper/1906.08885/) (2019)
- [Learning the Curriculum with Bayesian Optimization for Task-Specific Word Representation Learning](https://mlsanity.com/paper/1605.03852/) (2016)
- [Reinforced Curriculum Learning on Pre-trained Neural Machine Translation Models](https://mlsanity.com/paper/2004.05757/) (2020)
- [Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translation](https://mlsanity.com/paper/2010.02552/) (2020)
- [Dynamic Curriculum Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2011.14608/) (2020)
- [Self-Guided Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/2105.04475/) (2021)
- [Gradient-guided Loss Masking for Neural Machine Translation](https://mlsanity.com/paper/2102.13549/) (2021)

## Built on by (most cited first, 11 of 11)

- [On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.03154/) (2022)
- [Error Norm Truncation: Robust Training in the Presence of Data Noise for Text Generation Models](https://mlsanity.com/paper/2310.00840/) (2023)
- [A Psychology-based Unified Dynamic Framework for Curriculum Learning](https://mlsanity.com/paper/2408.05326/) (2024)
- [Let's Be Self-generated via Step by Step: A Curriculum Learning Approach to Automated Reasoning with Large Language Models](https://mlsanity.com/paper/2410.21728/) (2024)
- [RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translation](https://mlsanity.com/paper/2506.05070/) (2025)
- [Ling-CL: Understanding NLP Models through Linguistic Curricula](https://mlsanity.com/paper/2310.20121/) (2023)
- [Towards Inducing Long-Context Abilities in Multilingual Neural Machine Translation Models](https://mlsanity.com/paper/2408.11382/) (2024)
- [Beyond Shallow Heuristics: Leveraging Human Intuition for Curriculum Learning](https://mlsanity.com/paper/2508.19873/) (2025)
- [An Effective Incorporating Heterogeneous Knowledge Curriculum Learning for Sequence Labeling](https://mlsanity.com/paper/2402.13534/) (2024)
- [LexMatcher: Dictionary-centric Data Collection for LLM-based Machine Translation](https://mlsanity.com/paper/2406.01441/) (2024)
- [Testing the Assumptions of Active Learning for Translation Tasks with Few Samples](https://mlsanity.com/paper/2604.08977/) (2026)

## Abstract (arXiv)

Neural Machine Translation (NMT) models are typically trained on heterogeneous data that are concatenated and randomly shuffled. However, not all of the training data are equally useful to the model. Curriculum training aims to present the data to the NMT models in a meaningful order. In this work, we introduce a two-stage curriculum training framework for NMT where we fine-tune a base NMT model on subsets of data, selected by both deterministic scoring using pre-trained methods and online scoring that considers prediction scores of the emerging NMT model. Through comprehensive experiments on six language pairs comprising low- and high-resource languages from WMT'21, we have shown that our curriculum strategies consistently demonstrate better quality (up to +2.2 BLEU improvement) and faster convergence (approximately 50% fewer updates).