# Non-Parametric Adaptation for Neural Machine Translation

arXiv: https://arxiv.org/abs/1903.00058 · HTML: https://mlsanity.com/paper/1903.00058/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2019.

## Built on by (most cited first, 30 of 34)

- [A Survey on Retrieval-Augmented Text Generation](https://mlsanity.com/paper/2202.01110/) (2022)
- [Adaptive Semiparametric Language Models](https://mlsanity.com/paper/2102.02557/) (2021)
- [Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey](https://mlsanity.com/paper/2210.07700/) (2022)
- [A Benchmark for Learning to Translate a New Language from One Grammar Book](https://mlsanity.com/paper/2309.16575/) (2023)
- [Adaptive Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2105.13022/) (2021)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [Fast Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2105.14528/) (2021)
- [Transformer Based Deliberation for Two-Pass Speech Recognition](https://mlsanity.com/paper/2101.11577/) (2021)
- [Learning Kernel-Smoothed Machine Translation with Retrieved Examples](https://mlsanity.com/paper/2109.09991/) (2021)
- [Relational Memory Augmented Language Models](https://mlsanity.com/paper/2201.09680/) (2022)
- [Chunk-based Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2205.12230/) (2022)
- [Non-Parametric Unsupervised Domain Adaptation for Neural Machine Translation](https://mlsanity.com/paper/2109.06604/) (2021)
- [RETRONLU: Retrieval Augmented Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2109.10410/) (2021)
- [Towards Robust k-Nearest-Neighbor Machine Translation](https://mlsanity.com/paper/2210.08808/) (2022)
- [Non-Parametric Domain Adaptation for End-to-End Speech Translation](https://mlsanity.com/paper/2205.11211/) (2022)
- [Simple and Scalable Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2302.12188/) (2023)
- [Improving Automated Program Repair with Domain Adaptation](https://mlsanity.com/paper/2212.11414/) (2022)
- [Non-Parametric Online Learning from Human Feedback for Neural Machine Translation](https://mlsanity.com/paper/2109.11136/) (2021)
- [Faster Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2112.08152/) (2021)
- [Efficient Machine Translation Domain Adaptation](https://mlsanity.com/paper/2204.12608/) (2022)
- [Do Multilingual Neural Machine Translation Models Contain Language Pair Specific Attention Heads?](https://mlsanity.com/paper/2105.14940/) (2021)
- [Continual Learning of Neural Machine Translation within Low Forgetting Risk Regions](https://mlsanity.com/paper/2211.01542/) (2022)
- [Multi-Variate Time Series Forecasting on Variable Subsets](https://mlsanity.com/paper/2206.12626/) (2022)
- [Data-to-text Generation by Splicing Together Nearest Neighbors](https://mlsanity.com/paper/2101.08248/) (2021)
- [Self-generated Replay Memories for Continual Neural Machine Translation](https://mlsanity.com/paper/2403.13130/) (2024)
- [Multilingual k-Nearest-Neighbor Machine Translation](https://mlsanity.com/paper/2310.14644/) (2023)
- [Categorizing Semantic Representations for Neural Machine Translation](https://mlsanity.com/paper/2210.06709/) (2022)
- [Better Datastore, Better Translation: Generating Datastores from Pre-Trained Models for Nearest Neural Machine Translation](https://mlsanity.com/paper/2212.08822/) (2022)
- [Prompting Neural Machine Translation with Translation Memories](https://mlsanity.com/paper/2301.05380/) (2023)
- [N-Gram Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2301.12866/) (2023)

## Abstract (arXiv)

Neural Networks trained with gradient descent are known to be susceptible to catastrophic forgetting caused by parameter shift during the training process. In the context of Neural Machine Translation (NMT) this results in poor performance on heterogeneous datasets and on sub-tasks like rare phrase translation. On the other hand, non-parametric approaches are immune to forgetting, perfectly complementing the generalization ability of NMT. However, attempts to combine non-parametric or retrieval based approaches with NMT have only been successful on narrow domains, possibly due to over-reliance on sentence level retrieval. We propose a novel n-gram level retrieval approach that relies on local phrase level similarities, allowing us to retrieve neighbors that are useful for translation even when overall sentence similarity is low. We complement this with an expressive neural network, allowing our model to extract information from the noisy retrieved context. We evaluate our semi-parametric NMT approach on a heterogeneous dataset composed of WMT, IWSLT, JRC-Acquis and OpenSubtitles, and demonstrate gains on all 4 evaluation sets. The semi-parametric nature of our approach opens the door for non-parametric domain adaptation, demonstrating strong inference-time adaptation performance on new domains without the need for any parameter updates.