# Regularization techniques for fine-tuning in neural machine translation

arXiv: https://arxiv.org/abs/1707.09920 · HTML: https://mlsanity.com/paper/1707.09920/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jul 2017.

## Built on by (most cited first, 27 of 27)

- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [Taxonomizing local versus global structure in neural network loss landscapes](https://mlsanity.com/paper/2107.11228/) (2021)
- [Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guarantees](https://mlsanity.com/paper/2206.02659/) (2022)
- [Randomized Sparse Neural Galerkin Schemes for Solving Evolution Equations with Deep Networks](https://mlsanity.com/paper/2310.04867/) (2023)
- [How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizes](https://mlsanity.com/paper/2409.03454/) (2024)
- [Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation](https://mlsanity.com/paper/2103.13678/) (2021)
- [The Trade-offs of Domain Adaptation for Neural Language Models](https://mlsanity.com/paper/2109.10274/) (2021)
- [MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning](https://mlsanity.com/paper/2407.20999/) (2024)
- [Nearest Neighbor Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2205.00479/) (2022)
- [Source-Free Domain Adaptation for YOLO Object Detection](https://mlsanity.com/paper/2409.16538/) (2024)
- [Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?](https://mlsanity.com/paper/2410.09644/) (2024)
- [Entailed Between the Lines: Incorporating Implication into NLI](https://mlsanity.com/paper/2501.07719/) (2025)
- [Few-shot learning through contextual data augmentation](https://mlsanity.com/paper/2103.16911/) (2021)
- [When does Parameter-Efficient Transfer Learning Work for Machine Translation?](https://mlsanity.com/paper/2205.11277/) (2022)
- [Optimization Techniques for Sentiment Analysis Based on LLM (GPT-3)](https://mlsanity.com/paper/2405.09770/) (2024)
- [Continual Learning of Neural Machine Translation within Low Forgetting Risk Regions](https://mlsanity.com/paper/2211.01542/) (2022)
- [Task-adaptive Pre-training of Language Models with Word Embedding Regularization](https://mlsanity.com/paper/2109.08354/) (2021)
- [Life-long Learning for Multilingual Neural Machine Translation with Knowledge Distillation](https://mlsanity.com/paper/2212.02800/) (2022)
- [The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation](https://mlsanity.com/paper/2305.09652/) (2023)
- [Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation](https://mlsanity.com/paper/2404.04212/) (2024)
- [Addressing Bias Through Ensemble Learning and Regularized Fine-Tuning](https://mlsanity.com/paper/2402.00910/) (2024)
- [Domain adapted machine translation: What does catastrophic forgetting forget and why?](https://mlsanity.com/paper/2412.17537/) (2024)
- [Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translation](https://mlsanity.com/paper/2109.06308/) (2021)
- [Prompt-Based Tuning of Transformer Models for Multi-Center Medical Image Segmentation of Head and Neck Cancer](https://mlsanity.com/paper/2305.18948/) (2023)
- [Domain Specific Sub-network for Multi-Domain Neural Machine Translation](https://mlsanity.com/paper/2210.09805/) (2022)
- [Bridging the Domain Gaps in Context Representations for k-Nearest Neighbor Neural Machine Translation](https://mlsanity.com/paper/2305.16599/) (2023)
- [Reducing Bias in Pre-trained Models by Tuning while Penalizing Change](https://mlsanity.com/paper/2404.12292/) (2024)

## Abstract (arXiv)

We investigate techniques for supervised domain adaptation for neural machine translation where an existing model trained on a large out-of-domain dataset is adapted to a small in-domain dataset. In this scenario, overfitting is a major challenge. We investigate a number of techniques to reduce overfitting and improve transfer learning, including regularization techniques such as dropout and L2-regularization towards an out-of-domain prior. In addition, we introduce tuneout, a novel regularization technique inspired by dropout. We apply these techniques, alone and in combination, to neural machine translation, obtaining improvements on IWSLT datasets for English->German and English->Russian. We also investigate the amounts of in-domain training data needed for domain adaptation in NMT, and find a logarithmic relationship between the amount of training data and gain in BLEU score.