# ConsistTL: Modeling Consistency in Transfer Learning for Low-Resource Neural Machine Translation

arXiv: https://arxiv.org/abs/2212.04262 · HTML: https://mlsanity.com/paper/2212.04262/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 61 references that are in the map.
- First posted on arXiv 8 Dec 2022.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [Analyzing Uncertainty in Neural Machine Translation](https://mlsanity.com/paper/1803.00047/) (2018)
- [Meta-Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1808.08437/) (2018)
- [Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling](https://mlsanity.com/paper/1611.01462/) (2016)
- [Rapid Adaptation of Neural Machine Translation to New Languages](https://mlsanity.com/paper/1808.04189/) (2018)
- [Comparison of Diverse Decoding Methods from Conditional Language Models](https://mlsanity.com/paper/1906.06362/) (2019)
- [Semi-Supervised Sequence Modeling with Cross-View Training](https://mlsanity.com/paper/1809.08370/) (2018)
- [A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation](https://mlsanity.com/paper/2009.13818/) (2020)
- [Universal Neural Machine Translation for Extremely Low Resource Languages](https://mlsanity.com/paper/1802.05368/) (2018)
- [Revisiting Low-Resource Neural Machine Translation: A Case Study](https://mlsanity.com/paper/1905.11901/) (2019)
- [On the Inference Calibration of Neural Machine Translation](https://mlsanity.com/paper/2005.00963/) (2020)
- [Norm-Based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/2006.02014/) (2020)
- [Trivial Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1809.00357/) (2018)
- [Language Model Prior for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2004.14928/) (2020)
- [HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization](https://mlsanity.com/paper/2106.00149/) (2021)
- [Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translation](https://mlsanity.com/paper/2103.02262/) (2021)
- [Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies](https://mlsanity.com/paper/1905.05475/) (2019)
- [A simple but tough-to-beat baseline for the Fake News Challenge stance detection task](https://mlsanity.com/paper/1707.03264/) (2017)

## Built on by (most cited first, 4 of 4)

- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [Interactive Natural Language Processing](https://mlsanity.com/paper/2305.13246/) (2023)
- [Investigating Neural Machine Translation for Low-Resource Languages: Using Bavarian as a Case Study](https://mlsanity.com/paper/2404.08259/) (2024)
- [TACOS: Open Tagging and Comparative Scoring for Instruction Fine-Tuning Data Selection](https://mlsanity.com/paper/2507.03673/) (2025)

## Abstract (arXiv)

Transfer learning is a simple and powerful method that can be used to boost model performance of low-resource neural machine translation (NMT). Existing transfer learning methods for NMT are static, which simply transfer knowledge from a parent model to a child model once via parameter initialization. In this paper, we propose a novel transfer learning method for NMT, namely ConsistTL, which can continuously transfer knowledge from the parent model during the training of the child model. Specifically, for each training instance of the child model, ConsistTL constructs the semantically-equivalent instance for the parent model and encourages prediction consistency between the parent and child for this instance, which is equivalent to the child model learning each instance under the guidance of the parent model. Experimental results on five low-resource NMT tasks demonstrate that ConsistTL results in significant improvements over strong transfer learning baselines, with a gain up to 1.7 BLEU over the existing back-translation model on the widely-used WMT17 Turkish-English benchmark. Further analysis reveals that ConsistTL can improve the inference calibration of the child model. Code and scripts are freely available at https://github.com/NLP2CT/ConsistTL.