# Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation

arXiv: https://arxiv.org/abs/2004.14911 · HTML: https://mlsanity.com/paper/2004.14911/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 28 of 28)

- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Lightweight Adapter Tuning for Multilingual Speech Translation](https://mlsanity.com/paper/2106.01463/) (2021)
- [MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators](https://mlsanity.com/paper/2110.06609/) (2021)
- [Efficient Hierarchical Domain Adaptation for Pretrained Language Models](https://mlsanity.com/paper/2112.08786/) (2021)
- [Pretrained Models for Multilingual Federated Learning](https://mlsanity.com/paper/2206.02291/) (2022)
- [Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2105.03953/) (2021)
- [Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization](https://mlsanity.com/paper/2311.09344/) (2023)
- [Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders](https://mlsanity.com/paper/2104.08757/) (2021)
- [Multilingual Unsupervised Neural Machine Translation with Denoising Adapters](https://mlsanity.com/paper/2110.10472/) (2021)
- [Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters](https://mlsanity.com/paper/2110.09574/) (2021)
- [Language-Family Adapters for Low-Resource Multilingual Neural Machine Translation](https://mlsanity.com/paper/2209.15236/) (2022)
- [Multilingual Pre-training with Language and Task Adaptation for Multilingual Text Style Transfer](https://mlsanity.com/paper/2203.08552/) (2022)
- [Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adapter](https://mlsanity.com/paper/2305.12449/) (2023)
- [When does Parameter-Efficient Transfer Learning Work for Machine Translation?](https://mlsanity.com/paper/2205.11277/) (2022)
- [MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcycling](https://mlsanity.com/paper/2503.11144/) (2025)
- [Soft Language Clustering for Multilingual Model Pre-training](https://mlsanity.com/paper/2306.07610/) (2023)
- [KIT's Low-resource Speech Translation Systems for IWSLT2025: System Enhancement with Synthetic Data and Model Regularization](https://mlsanity.com/paper/2505.19679/) (2025)
- [m4Adapter: Multilingual Multi-Domain Adaptation for Machine Translation with a Meta-Adapter](https://mlsanity.com/paper/2210.11912/) (2022)
- [Enhanced Simultaneous Machine Translation with Word-level Policies](https://mlsanity.com/paper/2310.16417/) (2023)
- [From N-grams to Pre-trained Multilingual Models For Language Identification](https://mlsanity.com/paper/2410.08728/) (2024)
- [Extrapolating Multilingual Understanding Models as Multilingual Generators](https://mlsanity.com/paper/2305.13140/) (2023)
- [Mitigating Language-Level Performance Disparity in mPLMs via Teacher Language Selection and Cross-lingual Self-Distillation](https://mlsanity.com/paper/2404.08491/) (2024)
- [From Priest to Doctor: Domain Adaptation for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2412.00966/) (2024)
- [More Parameters? No Thanks!](https://mlsanity.com/paper/2107.09622/) (2021)
- [Thread-Efficient Decoding for Neural Texture Compression](https://mlsanity.com/paper/2608.27888/) (2026)

## Abstract (arXiv)

There has been recent success in pre-training on monolingual data and fine-tuning on Machine Translation (MT), but it remains unclear how to best leverage a pre-trained model for a given MT task. This paper investigates the benefits and drawbacks of freezing parameters, and adding new ones, when fine-tuning a pre-trained model on MT. We focus on 1) Fine-tuning a model trained only on English monolingual data, BART. 2) Fine-tuning a model trained on monolingual data from 25 languages, mBART. For BART we get the best performance by freezing most of the model parameters, and adding extra positional embeddings. For mBART we match or outperform the performance of naive fine-tuning for most language pairs with the encoder, and most of the decoder, frozen. The encoder-decoder attention parameters are most important to fine-tune. When constraining ourselves to an out-of-domain training set for Vietnamese to English we see the largest improvements over the fine-tuning baseline.