# Norm-Based Curriculum Learning for Neural Machine Translation

arXiv: https://arxiv.org/abs/2006.02014 · HTML: https://mlsanity.com/paper/2006.02014/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2020.

## Built on by (most cited first, 30 of 60)

- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Competence-based Multimodal Curriculum Learning for Medical Report Generation](https://mlsanity.com/paper/2206.14579/) (2022)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [Selective Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2105.12967/) (2021)
- [Preview, Attend and Review: Schema-Aware Curriculum Learning for Multi-Domain Dialog State Tracking](https://mlsanity.com/paper/2106.00291/) (2021)
- [Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learning](https://mlsanity.com/paper/2102.03554/) (2021)
- [A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation](https://mlsanity.com/paper/2203.01670/) (2022)
- [Deep Spatially and Temporally Aware Similarity Computation for Road Network Constrained Trajectories](https://mlsanity.com/paper/2112.09339/) (2021)
- [Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translation](https://mlsanity.com/paper/2103.02262/) (2021)
- [Progressive Multi-Granularity Training for Non-Autoregressive Translation](https://mlsanity.com/paper/2106.05546/) (2021)
- [Hybrid Curriculum Learning for Emotion Recognition in Conversation](https://mlsanity.com/paper/2112.11718/) (2021)
- [An Imitation Learning Curriculum for Text Editing with Non-Autoregressive Models](https://mlsanity.com/paper/2203.09486/) (2022)
- [Improving Neural Machine Translation by Bidirectional Training](https://mlsanity.com/paper/2109.07780/) (2021)
- [Wait-info Policy: Balancing Source and Target at Information Level for Simultaneous Machine Translation](https://mlsanity.com/paper/2210.11220/) (2022)
- [Boost Your Human Image Generation Model via Direct Preference Optimization](https://mlsanity.com/paper/2405.20216/) (2024)
- [On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation](https://mlsanity.com/paper/2110.01811/) (2021)
- [Improving Simultaneous Machine Translation with Monolingual Data](https://mlsanity.com/paper/2212.01188/) (2022)
- [A Contrastive Cross-Channel Data Augmentation Framework for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2204.07832/) (2022)
- [From Easy to Hard: Progressive Active Learning Framework for Infrared Small Target Detection with Single Point Supervision](https://mlsanity.com/paper/2412.11154/) (2024)
- [The USYD-JD Speech Translation System for IWSLT 2021](https://mlsanity.com/paper/2107.11572/) (2021)
- [Data Selection Curriculum for Neural Machine Translation](https://mlsanity.com/paper/2203.13867/) (2022)
- [Self-Guided Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/2105.04475/) (2021)
- [Vega-MT: The JD Explore Academy Translation System for WMT22](https://mlsanity.com/paper/2209.09444/) (2022)
- [Density-Based Dynamic Curriculum Learning for Intent Detection](https://mlsanity.com/paper/2108.10674/) (2021)
- [Difficulty-Aware Machine Translation Evaluation](https://mlsanity.com/paper/2107.14402/) (2021)
- [Exploiting Curriculum Learning in Unsupervised Neural Machine Translation](https://mlsanity.com/paper/2109.11177/) (2021)
- [Breaking the Representation Bottleneck of Chinese Characters: Neural Machine Translation with Stroke Sequence Modeling](https://mlsanity.com/paper/2211.12781/) (2022)
- [M2DF: Multi-grained Multi-curriculum Denoising Framework for Multimodal Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2310.14605/) (2023)
- [Estimating Machine Translation Difficulty](https://mlsanity.com/paper/2508.10175/) (2025)
- [Training Dynamics for Curriculum Learning: A Study on Monolingual and Cross-lingual NLU](https://mlsanity.com/paper/2210.12499/) (2022)

## Abstract (arXiv)

A neural machine translation (NMT) system is expensive to train, especially with high-resource settings. As the NMT architectures become deeper and wider, this issue gets worse and worse. In this paper, we aim to improve the efficiency of training an NMT by introducing a novel norm-based curriculum learning method. We use the norm (aka length or module) of a word embedding as a measure of 1) the difficulty of the sentence, 2) the competence of the model, and 3) the weight of the sentence. The norm-based sentence difficulty takes the advantages of both linguistically motivated and model-based sentence difficulties. It is easy to determine and contains learning-dependent features. The norm-based model competence makes NMT learn the curriculum in a fully automated way, while the norm-based sentence weight further enhances the learning of the vector representation of the NMT. Experimental results for the WMT'14 English-German and WMT'17 Chinese-English translation tasks demonstrate that the proposed method outperforms strong baselines in terms of BLEU score (+1.17/+1.56) and training speedup (2.22x/3.33x).