mlsanityData to 7 Oct 2026

Generalization through Memorization: Nearest Neighbor Language Models

arXiv 1911.00172 ↗First posted 1 Nov 2019cs.CLcited by 788

Not on the map. Why?

Cited by 208 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce kNN-LMs, which extend a pre-trained neural language model (LM) by linearly interpolating it with a k-nearest neighbors (kNN) model. The nearest neighbors are computed according to distance in the pre-trained LM embedding space, and can be drawn from any text collection, including the original LM training data. Applying this augmentation to a strong Wikitext-103 LM, with neighbors drawn from the original training set, our kNN-LM achieves a new state-of-the-art perplexity of 15.79 - a 2.9 point improvement with no additional training. We also show that this approach has implications for efficiently scaling up to larger training sets and allows for effective domain adaptation, by simply varying the nearest neighbor datastore, again without further training. Qualitatively, the model is particularly helpful in predicting rare patterns, such as factual knowledge. Together, these results strongly suggest that learning similarity between sequences of text is easier than predicting the next word, and that nearest neighbor search is an effective approach for language modeling in the long tail.

Key facts. Cited by 788 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 71; 2022: 97; 2023: 160; 2024: 207; 2025: 120; 2026: 131.

788 papers.

Sort:

By area:

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  8. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  10. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  11. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  12. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
38 more
  1. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  2. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  3. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  4. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  5. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  6. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  7. 2023Active Retrieval Augmented Generationcited by 506
  8. 2022Memory-Based Model Editing at Scalecited by 443
  9. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  10. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  11. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  12. 2023Augmented Language Models: a Surveycited by 398
  13. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  14. 2024The Faiss librarycited by 370
  15. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  16. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  17. 2023Adapting Language Models to Compress Contextscited by 350
  18. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  19. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  20. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  21. 2024Titans: Learning to Memorize at Test Timecited by 303
  22. 2023Tool Learning with Foundation Modelscited by 302
  23. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  24. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  25. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  26. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  27. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  28. 2022A Contrastive Framework for Neural Text Generationcited by 224
  29. 2022Memorizing Transformerscited by 219
  30. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  31. 2021Internet-Augmented Dialogue Generationcited by 217
  32. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  33. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  34. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
  35. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  36. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  37. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  38. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 194

50 most cited of 788 All 788 (JSON)