mlsanityData to 7 Oct 2026

Efficient Nearest Neighbor Language Models

arXiv 2109.04212 ↗First posted 9 Sep 2021cs.CLcited by 93

Not on the map. Why?

Cited by 31 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Non-parametric neural language models (NLMs) learn predictive distributions of text utilizing an external datastore, which allows them to learn through explicitly memorizing the training datapoints. While effective, these models often require retrieval from a large datastore at test time, significantly increasing the inference overhead and thus limiting the deployment of non-parametric NLMs in practical applications. In this paper, we take the recently proposed k-nearest neighbors language model (Khandelwal et al., 2020) as an example, exploring methods to improve its efficiency along various dimensions. Experiments on the standard WikiText-103 benchmark and domain-adaptation datasets show that our methods are able to achieve up to a 6x speed-up in inference speed while retaining comparable performance. The empirical analysis we present may provide guidelines for future research seeking to develop or deploy more efficient non-parametric NLMs.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Builds on 23 of its 35 references that are in the map. First posted on arXiv 9 Sep 2021.

Builds on

23 of 35 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2017Billion-scale similarity search with GPUscited by 2,477
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  11. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  12. 2017Six Challenges for Neural Machine Translationcited by 334
11 more
  1. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  2. 2020Nearest Neighbor Machine Translationcited by 209
  3. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  4. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  5. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  6. 2017Generating Sentences by Editing Prototypescited by 88
  7. 2021Adaptive Semiparametric Language Modelscited by 83
  8. 2021NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learnedcited by 52
  9. 2020Learning Sparse Prototypes for Text Generationcited by 15
  10. 2018Quicker ADC : Unlocking the hidden potential of Product Quantization with SIMDcited by 14
  11. 2017Learning when to skim and when to readcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 19; 2023: 26; 2024: 23; 2025: 16; 2026: 7.

93 papers.

Sort:

By area:

  1. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  2. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  3. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  4. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  5. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  6. 2022Training Language Models with Memory Augmentationcited by 112
  7. 2021The Efficiency Misnomercited by 98
  8. 2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 82
  9. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  10. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  11. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  12. 2022Neuro-Symbolic Language Modeling with Automaton-augmented Retrievalcited by 60
38 more
  1. 2024Repoformer: Selective Retrieval for Repository-Level Code Completioncited by 55
  2. 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 54
  3. 2022Nonparametric Masked Language Modelingcited by 39
  4. 2022Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learningcited by 36
  5. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  6. 2024RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewardscited by 33
  7. 2023Privacy Implications of Retrieval-Based Language Modelscited by 31
  8. 2021How BPE Affects Memorization in Transformerscited by 29
  9. 2023Why do Nearest Neighbor Language Models Work?cited by 29
  10. 2025Adaptive Tool Use in Large Language Models with Meta-Cognition Triggercited by 29
  11. 2025Metacognitive Reuse: Turning Recurring LLM Reasoning Into Concise Behaviorscited by 28
  12. 2022kNN-Prompt: Nearest Neighbor Zero-Shot Inferencecited by 27
  13. 2024Metacognitive Retrieval-Augmented Large Language Modelscited by 26
  14. 2024Nearest Neighbor Speculative Decoding for LLM Generation and Attributioncited by 26
  15. 2023Copy Is All You Needcited by 25
  16. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processingcited by 25
  17. 2022Efficient Cluster-Based k-Nearest-Neighbor Machine Translationcited by 24
  18. 2022Chunk-based Nearest Neighbor Machine Translationcited by 21
  19. 2024Non-Exchangeable Conformal Language Generation with Nearest Neighborscited by 21
  20. 2024Accelerating Retrieval-Augmented Language Model Serving with Speculationcited by 20
  21. 2025MLP Memory: A Retriever-Pretrained Memory for Large Language Modelscited by 19
  22. 2022You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LMcited by 18
  23. 2025Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Modelscited by 16
  24. 2023Retrosynthesis Prediction with Local Template Retrievalcited by 15
  25. 2024On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domainscited by 14
  26. 2022Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt Tuningcited by 13
  27. 2023kNN-Adapter: Efficient Domain Adaptation for Black-Box Language Modelscited by 13
  28. 2024Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoningcited by 13
  29. 2022Reasoning Through Memorization: Nearest Neighbor Knowledge Graph Embeddingscited by 12
  30. 2022Non-Parametric Domain Adaptation for End-to-End Speech Translationcited by 12
  31. 2023Simple and Scalable Nearest Neighbor Machine Translationcited by 12
  32. 2023Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memorycited by 12
  33. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  34. 2023KNN-LM Does Not Improve Open-ended Text Generationcited by 10
  35. 2024FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval-Augmented Code Completioncited by 9
  36. 2025Self-Routing RAG: Binding Selective Retrieval with Knowledge Verbalizationcited by 9
  37. 2025Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Modelscited by 8
  38. 2022Efficient Machine Translation Domain Adaptationcited by 7

50 most cited of 93 All 93 (JSON)