# Nearest Neighbor Non-autoregressive Text Generation

arXiv: https://arxiv.org/abs/2208.12496 · HTML: https://mlsanity.com/paper/2208.12496/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 26 Aug 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2010.00710/) (2020)
- [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [Insertion Transformer: Flexible Sequence Generation via Insertion Operations](https://mlsanity.com/paper/1902.03249/) (2019)
- [Glancing Transformer for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2008.07905/) (2020)
- [Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade](https://mlsanity.com/paper/2012.15833/) (2020)
- [Neural Machine Translation with Monolingual Translation Memory](https://mlsanity.com/paper/2105.11269/) (2021)
- [Fast Structured Decoding for Sequence Models](https://mlsanity.com/paper/1910.11555/) (2019)
- [Adaptive Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2105.13022/) (2021)
- [Text Generation with Exemplar-based Adaptive Decoding](https://mlsanity.com/paper/1904.04428/) (2019)
- [EDITOR: an Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints](https://mlsanity.com/paper/2011.06868/) (2020)
- [Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2009.07177/) (2020)
- [Data-to-text Generation by Splicing Together Nearest Neighbors](https://mlsanity.com/paper/2101.08248/) (2021)

## Built on by (most cited first, 7 of 7)

- [Retrieving Examples from Memory for Retrieval Augmented Neural Machine Translation: A Systematic Comparison](https://mlsanity.com/paper/2404.02835/) (2024)
- [Integrating Translation Memories into Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2210.06020/) (2022)
- [Towards Example-Based NMT with Multi-Levenshtein Transformers](https://mlsanity.com/paper/2310.08967/) (2023)
- [Optimizing example selection for retrieval-augmented machine translation with translation memories](https://mlsanity.com/paper/2405.15070/) (2024)
- [Non-autoregressive Personalized Bundle Generation](https://mlsanity.com/paper/2406.06925/) (2024)
- [Improving Retrieval-Augmented Neural Machine Translation with Monolingual Data](https://mlsanity.com/paper/2504.21747/) (2025)
- [Reasoning about In-Context Samples for Machine-Translation](https://mlsanity.com/paper/2608.27036/) (2026)

## Abstract (arXiv)

Non-autoregressive (NAR) models can generate sentences with less computation than autoregressive models but sacrifice generation quality. Previous studies addressed this issue through iterative decoding. This study proposes using nearest neighbors as the initial state of an NAR decoder and editing them iteratively. We present a novel training strategy to learn the edit operations on neighbors to improve NAR text generation. Experimental results show that the proposed method (NeighborEdit) achieves higher translation quality (1.69 points higher than the vanilla Transformer) with fewer decoding iterations (one-eighteenth fewer iterations) on the JRC-Acquis En-De dataset, the common benchmark dataset for machine translation using nearest neighbors. We also confirm the effectiveness of the proposed method on a data-to-text task (WikiBio). In addition, the proposed method outperforms an NAR baseline on the WMT'14 En-De dataset. We also report analysis on neighbor examples used in the proposed method.