mlsanityData to 7 Oct 2026

KNN-LM Does Not Improve Open-ended Text Generation

arXiv 2305.14625 ↗First posted 24 May 2023cs.CLcited by 10

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

In this paper, we study the generation quality of interpolation-based retrieval-augmented language models (LMs). These methods, best exemplified by the KNN-LM, interpolate the LM's predicted distribution of the next word with a distribution formed from the most relevant retrievals for a given prefix. While the KNN-LM and related methods yield impressive decreases in perplexity, we discover that they do not exhibit corresponding improvements in open-ended generation quality, as measured by both automatic evaluation metrics (e.g., MAUVE) and human evaluations. Digging deeper, we find that interpolating with a retrieval distribution actually increases perplexity compared to a baseline Transformer LM for the majority of tokens in the WikiText-103 test set, even though the overall perplexity is lower due to a smaller number of tokens for which perplexity dramatically decreases after interpolation. However, when decoding a long sequence at inference time, significant improvements on this smaller subset of tokens are washed out by slightly worse predictions on most tokens. Furthermore, we discover that the entropy of the retrieval distribution increases faster than that of the base LM as the generated sequence becomes longer, which indicates that retrieval is less reliable when using model-generated text as queries (i.e., is subject to exposure bias). We hope that our analysis spurs future work on improved decoding algorithms and interpolation strategies for retrieval-augmented language models.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 40 of its 48 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

40 of 48 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  9. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  10. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  11. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  12. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
28 more
  1. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  3. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  4. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  5. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  6. 2020Evaluation of Text Generation: A Surveycited by 275
  7. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  8. 2022Memorizing Transformerscited by 219
  9. 2020Nearest Neighbor Machine Translationcited by 209
  10. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 200
  11. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  12. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  13. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  14. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  15. 2021Entity-level Factual Consistency of Abstractive Text Summarizationcited by 118
  16. 2022Training Language Models with Memory Augmentationcited by 112
  17. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  18. 2021Efficient Nearest Neighbor Language Modelscited by 93
  19. 2021Adaptive Semiparametric Language Modelscited by 83
  20. 2016Improving Neural Language Models with a Continuous Cachecited by 81
  21. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  22. 2022Neuro-Symbolic Language Modeling with Automaton-augmented Retrievalcited by 60
  23. 2022Retrieval-Enhanced Machine Learningcited by 46
  24. 2022Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literaturecited by 45
  25. 2019Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?cited by 33
  26. 2023Why do Nearest Neighbor Language Models Work?cited by 29
  27. 2022You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LMcited by 18
  28. 2022Generalization Properties of Retrieval-based Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 3; 2026: 1.

10 papers.

  1. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  2. 2023Understanding Retrieval Augmentation for Long-Form Question Answeringcited by 45
  3. 2024PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-designcited by 28
  4. 2024Nearest Neighbor Speculative Decoding for LLM Generation and Attributioncited by 26
  5. 2024Retrieval is Accurate Generationcited by 10
  6. 2024MoRSE: Bridging the Gap in Cybersecurity Expertise with Retrieval Augmented Generationcited by 8
  7. 2025Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarkscited by 8
  8. 2025DioR: Adaptive Cognitive Detection and Contextual Retrieval Optimization for Dynamic Retrieval-Augmented Generationcited by 7
  9. 2025CoCoLex: Confidence-guided Copy-based Decoding for Grounded Legal Text Generationcited by 6
  10. 2026Training-Free Token-Level Steering for LLM Personalized Co-Writingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales
  2. 2024A Theory for Token-Level Harmonization in Retrieval-Augmented Generation
  3. 2025LFD: Layer Fused Decoding to Exploit External Knowledge in Retrieval-Augmented Generation
  4. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  5. 2024E2E-AFG: An End-to-End Model with Adaptive Filtering for Retrieval-Augmented Generation
  6. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Models
  7. 2026Latent Abstraction for Retrieval-Augmented Generation
  8. 2023Benchmarking Large Language Models in Retrieval-Augmented Generation