# Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation

arXiv: https://arxiv.org/abs/2003.02877 · HTML: https://mlsanity.com/paper/2003.02877/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Mar 2020.

## Built on by (most cited first, 7 of 7)

- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [Datasheet for the Pile](https://mlsanity.com/paper/2201.07311/) (2022)
- [Fast Vocabulary Transfer for Language Model Compression](https://mlsanity.com/paper/2402.09977/) (2024)
- [Sampling and Filtering of Neural Machine Translation Distillation Data](https://mlsanity.com/paper/2104.00664/) (2021)
- [KD4MT: A Survey of Knowledge Distillation for Machine Translation](https://mlsanity.com/paper/2602.15845/) (2026)
- [Effective General-Domain Data Inclusion for the Machine Translation Task by Vanilla Transformers](https://mlsanity.com/paper/2209.14073/) (2022)
- [Stolen Subwords: Importance of Vocabularies for Machine Translation Model Stealing](https://mlsanity.com/paper/2401.16055/) (2024)

## Abstract (arXiv)

We explore best practices for training small, memory efficient machine translation models with sequence-level knowledge distillation in the domain adaptation setting. While both domain adaptation and knowledge distillation are widely-used, their interaction remains little understood. Our large-scale empirical results in machine translation (on three language pairs with three domains each) suggest distilling twice for best performance: once using general-domain data and again using in-domain data with an adapted teacher.