# Exploring Benefits of Transfer Learning in Neural Machine Translation

arXiv: https://arxiv.org/abs/2001.01622 · HTML: https://mlsanity.com/paper/2001.01622/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jan 2020.

## Built on by (most cited first, 5 of 5)

- [Neural Supervised Domain Adaptation by Augmenting Pre-trained Models with Random Units](https://mlsanity.com/paper/2106.04935/) (2021)
- [Efficiently Upgrading Multilingual Machine Translation Models to Support More Languages](https://mlsanity.com/paper/2302.03528/) (2023)
- [From Zero to Production: Baltic-Ukrainian Machine Translation Systems to Aid Refugees](https://mlsanity.com/paper/2209.14142/) (2022)
- [Massively Multilingual Text Translation For Low-Resource Languages](https://mlsanity.com/paper/2401.16582/) (2024)
- [Cross-Lingual Transfer for Machine Translation in Turkic Languages](https://mlsanity.com/paper/2607.29355/) (2026)

## Abstract (arXiv)

Neural machine translation is known to require large numbers of parallel training sentences, which generally prevent it from excelling on low-resource language pairs. This thesis explores the use of cross-lingual transfer learning on neural networks as a way of solving the problem with the lack of resources. We propose several transfer learning approaches to reuse a model pretrained on a high-resource language pair. We pay particular attention to the simplicity of the techniques. We study two scenarios: (a) when we reuse the high-resource model without any prior modifications to its training process and (b) when we can prepare the first-stage high-resource model for transfer learning in advance. For the former scenario, we present a proof-of-concept method by reusing a model trained by other researchers. In the latter scenario, we present a method which reaches even larger improvements in translation performance. Apart from proposed techniques, we focus on an in-depth analysis of transfer learning techniques and try to shed some light on transfer learning improvements. We show how our techniques address specific problems of low-resource languages and are suitable even in high-resource transfer learning. We evaluate the potential drawbacks and behavior by studying transfer learning in various situations, for example, under artificially damaged training corpora, or with fixed various model parts.