# Adaptive Nearest Neighbor Machine Translation

arXiv: https://arxiv.org/abs/2105.13022 · HTML: https://mlsanity.com/paper/2105.13022/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 20 references that are in the map.
- First posted on arXiv 27 May 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2010.00710/) (2020)
- [Facebook FAIR's WMT19 News Translation Task Submission](https://mlsanity.com/paper/1907.06616/) (2019)
- [Achieving Human Parity on Automatic Chinese to English News Translation](https://mlsanity.com/paper/1803.05567/) (2018)
- [Guiding Neural Machine Translation with Retrieved Translation Pieces](https://mlsanity.com/paper/1804.02559/) (2018)
- [BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA](https://mlsanity.com/paper/2005.00766/) (2020)
- [Non-Parametric Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1903.00058/) (2019)
- [Unsupervised Domain Adaptation for Neural Machine Translation with Domain-Aware Feature Embeddings](https://mlsanity.com/paper/1908.10430/) (2019)
- [Iterative Domain-Repaired Back-Translation](https://mlsanity.com/paper/2010.02473/) (2020)

## Built on by (most cited first, 30 of 58)

- [A Survey on Retrieval-Augmented Text Generation](https://mlsanity.com/paper/2202.01110/) (2022)
- [Training Language Models with Memory Augmentation](https://mlsanity.com/paper/2205.12674/) (2022)
- [Retrieval-Augmented Generation for Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.13193/) (2024)
- [Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse](https://mlsanity.com/paper/2409.11242/) (2024)
- [Efficient Cluster-Based k-Nearest-Neighbor Machine Translation](https://mlsanity.com/paper/2204.06175/) (2022)
- [Improving Few-Shot Performance of Language Models via Nearest Neighbor Calibration](https://mlsanity.com/paper/2212.02216/) (2022)
- [Chunk-based Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2205.12230/) (2022)
- [Non-Exchangeable Conformal Language Generation with Nearest Neighbors](https://mlsanity.com/paper/2402.00707/) (2024)
- [Non-Parametric Unsupervised Domain Adaptation for Neural Machine Translation](https://mlsanity.com/paper/2109.06604/) (2021)
- [Layer-wise Model Pruning based on Mutual Information](https://mlsanity.com/paper/2108.12594/) (2021)
- [You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LM](https://mlsanity.com/paper/2210.15859/) (2022)
- [Retrosynthesis Prediction with Local Template Retrieval](https://mlsanity.com/paper/2306.04123/) (2023)
- [Towards Robust k-Nearest-Neighbor Machine Translation](https://mlsanity.com/paper/2210.08808/) (2022)
- [kNN-Adapter: Efficient Domain Adaptation for Black-Box Language Models](https://mlsanity.com/paper/2302.10879/) (2023)
- [Non-Parametric Domain Adaptation for End-to-End Speech Translation](https://mlsanity.com/paper/2205.11211/) (2022)
- [Simple and Scalable Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2302.12188/) (2023)
- [Nearest Neighbor Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2205.00479/) (2022)
- [Non-Parametric Online Learning from Human Feedback for Neural Machine Translation](https://mlsanity.com/paper/2109.11136/) (2021)
- [Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks](https://mlsanity.com/paper/2401.10070/) (2024)
- [Efficient Machine Translation Domain Adaptation](https://mlsanity.com/paper/2204.12608/) (2022)
- [Nearest Neighbor Non-autoregressive Text Generation](https://mlsanity.com/paper/2208.12496/) (2022)
- [Improving Retrieval Augmented Neural Machine Translation by Controlling Source and Fuzzy-Match Interactions](https://mlsanity.com/paper/2210.05047/) (2022)
- [What Knowledge Is Needed? Towards Explainable Memory for kNN-MT Domain Adaptation](https://mlsanity.com/paper/2211.04052/) (2022)
- [KIT's Multilingual Speech Translation System for IWSLT 2023](https://mlsanity.com/paper/2306.05320/) (2023)
- [Tram: A Token-level Retrieval-augmented Mechanism for Source Code Summarization](https://mlsanity.com/paper/2305.11074/) (2023)
- [INK: Injecting kNN Knowledge in Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2306.06381/) (2023)
- [Integrating Translation Memories into Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2210.06020/) (2022)
- [Federated Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2302.12211/) (2023)
- [NNOSE: Nearest Neighbor Occupational Skill Extraction](https://mlsanity.com/paper/2401.17092/) (2024)
- [Mind Your Neighbours: Leveraging Analogous Instances for Rhetorical Role Labeling for Legal Documents](https://mlsanity.com/paper/2404.01344/) (2024)

## Abstract (arXiv)

kNN-MT, recently proposed by Khandelwal et al. (2020a), successfully combines pre-trained neural machine translation (NMT) model with token-level k-nearest-neighbor (kNN) retrieval to improve the translation accuracy. However, the traditional kNN algorithm used in kNN-MT simply retrieves a same number of nearest neighbors for each target token, which may cause prediction errors when the retrieved neighbors include noises. In this paper, we propose Adaptive kNN-MT to dynamically determine the number of k for each target token. We achieve this by introducing a light-weight Meta-k Network, which can be efficiently trained with only a few training samples. On four benchmark machine translation datasets, we demonstrate that the proposed method is able to effectively filter out the noises in retrieval results and significantly outperforms the vanilla kNN-MT model. Even more noteworthy is that the Meta-k Network learned on one domain could be directly applied to other domains and obtain consistent improvements, illustrating the generality of our method. Our implementation is open-sourced at https://github.com/zhengxxn/adaptive-knn-mt.