# Self-Guided Curriculum Learning for Neural Machine Translation

arXiv: https://arxiv.org/abs/2105.04475 · HTML: https://mlsanity.com/paper/2105.04475/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 10 May 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Competence-based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1903.09848/) (2019)
- [Achieving Human Parity on Automatic Chinese to English News Translation](https://mlsanity.com/paper/1803.05567/) (2018)
- [Curriculum Learning and Minibatch Bucketing in Neural Machine Translation](https://mlsanity.com/paper/1707.09533/) (2017)
- [Norm-Based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/2006.02014/) (2020)
- [Curriculum Learning for Domain Adaptation in Neural Machine Translation](https://mlsanity.com/paper/1905.05816/) (2019)
- [An Empirical Exploration of Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1811.00739/) (2018)
- [Understanding and Improving Lexical Choice in Non-Autoregressive Translation](https://mlsanity.com/paper/2012.14583/) (2020)
- [Visualizing and Understanding Curriculum Learning for Long Short-Term Memory Networks](https://mlsanity.com/paper/1611.06204/) (2016)
- [SlotRefine: A Fast Non-Autoregressive Model for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/2010.02693/) (2020)
- [Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translation](https://mlsanity.com/paper/2103.02262/) (2021)
- [Self-Attention with Cross-Lingual Position Representation](https://mlsanity.com/paper/2004.13310/) (2020)
- [Context-Aware Cross-Attention for Non-Autoregressive Translation](https://mlsanity.com/paper/2011.00770/) (2020)
- [Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning](https://mlsanity.com/paper/2012.14768/) (2020)
- [Dynamic Data Selection and Weighting for Iterative Back-Translation](https://mlsanity.com/paper/2004.03672/) (2020)
- [Dynamic Curriculum Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2011.14608/) (2020)
- [The University of Sydney's Machine Translation System for WMT19](https://mlsanity.com/paper/1907.00494/) (2019)
- [Zero-Shot Translation Quality Estimation with Explicit Cross-Lingual Patterns](https://mlsanity.com/paper/2010.04989/) (2020)

## Built on by (most cited first, 9 of 9)

- [Revisiting Catastrophic Forgetting in Large Language Model Tuning](https://mlsanity.com/paper/2406.04836/) (2024)
- [Progressive Multi-Granularity Training for Non-Autoregressive Translation](https://mlsanity.com/paper/2106.05546/) (2021)
- [Improving Neural Machine Translation by Bidirectional Training](https://mlsanity.com/paper/2109.07780/) (2021)
- [The USYD-JD Speech Translation System for IWSLT 2021](https://mlsanity.com/paper/2107.11572/) (2021)
- [Data Selection Curriculum for Neural Machine Translation](https://mlsanity.com/paper/2203.13867/) (2022)
- [Revisiting Multilingual Data Mixtures in Language Model Pretraining](https://mlsanity.com/paper/2510.25947/) (2025)
- [Vega-MT: The JD Explore Academy Translation System for WMT22](https://mlsanity.com/paper/2209.09444/) (2022)
- [Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap](https://mlsanity.com/paper/2508.04149/) (2025)
- [Improving Neural Machine Translation by Denoising Training](https://mlsanity.com/paper/2201.07365/) (2022)

## Abstract (arXiv)

In the field of machine learning, the well-trained model is assumed to be able to recover the training labels, i.e. the synthetic labels predicted by the model should be as close to the ground-truth labels as possible. Inspired by this, we propose a self-guided curriculum strategy to encourage the learning of neural machine translation (NMT) models to follow the above recovery criterion, where we cast the recovery degree of each training example as its learning difficulty. Specifically, we adopt the sentence level BLEU score as the proxy of recovery degree. Different from existing curricula relying on linguistic prior knowledge or third-party language models, our chosen learning difficulty is more suitable to measure the degree of knowledge mastery of the NMT models. Experiments on translation benchmarks, including WMT14 EnglishRightarrowGerman and WMT17 ChineseRightarrowEnglish, demonstrate that our approach can consistently improve translation performance against strong baseline Transformer.