# INK: Injecting kNN Knowledge in Nearest Neighbor Machine Translation

arXiv: https://arxiv.org/abs/2306.06381 · HTML: https://mlsanity.com/paper/2306.06381/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 10 Jun 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2010.00710/) (2020)
- [Facebook FAIR's WMT19 News Translation Task Submission](https://mlsanity.com/paper/1907.06616/) (2019)
- [Glancing Transformer for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2008.07905/) (2020)
- [Adaptive Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2105.13022/) (2021)
- [Contrastive Learning with Adversarial Perturbations for Conditional Text Generation](https://mlsanity.com/paper/2012.07280/) (2020)
- [Fast Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2105.14528/) (2021)
- [Efficient Cluster-Based k-Nearest-Neighbor Machine Translation](https://mlsanity.com/paper/2204.06175/) (2022)
- [Chunk-based Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2205.12230/) (2022)
- [CoNT: Contrastive Neural Text Generation](https://mlsanity.com/paper/2205.14690/) (2022)
- [textitlatent-GLAT: Glancing at Latent Variables for Parallel Text Generation](https://mlsanity.com/paper/2204.02030/) (2022)
- [Towards Robust k-Nearest-Neighbor Machine Translation](https://mlsanity.com/paper/2210.08808/) (2022)
- [The Volctrans GLAT System: Non-autoregressive Translation Meets WMT21](https://mlsanity.com/paper/2109.11247/) (2021)
- [Non-Autoregressive Machine Translation: It's Not as Fast as it Seems](https://mlsanity.com/paper/2205.01966/) (2022)
- [Nearest Neighbor Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2205.00479/) (2022)
- [Efficient Machine Translation Domain Adaptation](https://mlsanity.com/paper/2204.12608/) (2022)
- [Learning Decoupled Retrieval Representation for Nearest Neighbour Neural Machine Translation](https://mlsanity.com/paper/2209.08738/) (2022)
- [Better Datastore, Better Translation: Generating Datastores from Pre-Trained Models for Nearest Neural Machine Translation](https://mlsanity.com/paper/2212.08822/) (2022)

## Built on by (most cited first, 6 of 6)

- [Retrieval-Augmented Generation for Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.13193/) (2024)
- [Efficient k-Nearest-Neighbor Machine Translation with Dynamic Retrieval](https://mlsanity.com/paper/2406.06073/) (2024)
- [Efficient Nearest Neighbor based Uncertainty Estimation for Natural Language Processing Tasks](https://mlsanity.com/paper/2407.02138/) (2024)
- [QASE Enhanced PLMs: Improved Control in Text Generation for MRC](https://mlsanity.com/paper/2403.04771/) (2024)
- [Enhancing Pre-Trained Generative Language Models with Question Attended Span Extraction on Machine Reading Comprehension](https://mlsanity.com/paper/2404.17991/) (2024)
- [RASST: Retrieval-Augmented Simultaneous Speech Translation](https://mlsanity.com/paper/2601.22777/) (2026)

## Abstract (arXiv)

Neural machine translation has achieved promising results on many translation tasks. However, previous studies have shown that neural models induce a non-smooth representation space, which harms its generalization results. Recently, kNN-MT has provided an effective paradigm to smooth the prediction based on neighbor representations during inference. Despite promising results, kNN-MT usually requires large inference overhead. We propose an effective training framework INK to directly smooth the representation space via adjusting representations of kNN neighbors with a small number of new parameters. The new parameters are then used to refresh the whole representation datastore to get new kNN knowledge asynchronously. This loop keeps running until convergence. Experiments on four benchmark datasets show that method achieves average gains of 1.99 COMET and 1.0 BLEU, outperforming the state-of-the-art kNN-MT system with 0.02x memory space and 1.9x inference speedup.