mlsanityData to 14 Oct 2026

Nearest Neighbor Speculative Decoding for LLM Generation and Attribution

arXiv 2405.19325 ↗First posted 29 May 2024cs.CLcited by 25

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Large language models (LLMs) often hallucinate and lack the ability to provide attribution for their generations. Semi-parametric LMs, such as kNN-LM, approach these limitations by refining the output of an LM for a given prompt using its nearest neighbor matches in a non-parametric data store. However, these models often exhibit slow inference speeds and produce non-fluent texts. In this paper, we introduce Nearest Neighbor Speculative Decoding (NEST), a novel semi-parametric language modeling approach that is capable of incorporating real-world text spans of arbitrary length into the LM generations and providing attribution to their sources. NEST performs token-level retrieval at each inference step to compute a semi-parametric mixture distribution and identify promising span continuations in a corpus. It then uses an approximate speculative decoding procedure that accepts a prefix of the retrieved span or generates a new token. NEST significantly enhances the generation quality and attribution rate of the base LM across a variety of knowledge-intensive tasks, surpassing the conventional kNN-LM method and performing competitively with in-context retrieval augmentation. In addition, NEST substantially improves the generation speed, achieving a 1.8x speedup in inference time when applied to Llama-2-Chat 70B. Code will be released at https://github.com/facebookresearch/NEST/tree/main.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 52 of its 61 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

52 of 61 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  2. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  3. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  4. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  11. 2016Pointer Sentinel Mixture Modelscited by 3,096
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
36 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  3. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  6. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  7. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  8. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  9. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  10. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  11. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  12. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  13. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  14. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  15. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  16. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  17. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 195
  18. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  19. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  20. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 118
  21. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  22. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 97
  23. 2021Efficient Nearest Neighbor Language Modelscited by 93
  24. 2022PEER: A Collaborative Language Modelcited by 86
  25. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  26. 2022Generating Literal and Implied Subquestions to Fact-check Complex Claimscited by 66
  27. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 65
  28. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Studycited by 63
  29. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53
  30. 2021FRUIT: Faithfully Reflecting Updated Information in Textcited by 39
  31. 2023Copy Is All You Needcited by 27
  32. 2022Chunk-based Nearest Neighbor Machine Translationcited by 21
  33. 2022You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LMcited by 18
  34. 2023KNN-LM Does Not Improve Open-ended Text Generationcited by 10
  35. 2024Retrieval is Accurate Generationcited by 10
  36. 2022Adaptation Approaches for Nearest Neighbor Language Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 12; 2026: 6.

25 papers.

Sort:

By area:

  1. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  2. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  3. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  4. 2024A Tighter Complexity Analysis of SparseGPTcited by 19
  5. 2024Mixture of Attentions For Speculative Decodingcited by 17
  6. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 16
  7. 2025TokenShapley: Token Level Context Attribution with Shapley Valuecited by 15
  8. 2024Improving Factuality with Explicit Working Memorycited by 10
  9. 2025Speculate, then Collaborate: Fusing Knowledge of Language Models during Decodingcited by 8
  10. 2025Attribution, Citation, and Quotation: A Survey of Evidence-based Text Generation with Large Language Modelscited by 7
  11. 2025DReSD: Dense Retrieval for Speculative Decodingcited by 6
  12. 2025VIBE: Vector Index Benchmark for Embeddingscited by 6
13 more
  1. 2025A Survey on Proactive Defense Strategies Against Misinformation in Large Language Modelscited by 6
  2. 2025Large Language Model Sourcing: A Surveycited by 5
  3. 2025TokenSwift: Lossless Acceleration of Ultra Long Sequence Generationcited by 4
  4. 2025Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMscited by 4
  5. 2025OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device Speculative Decodingcited by 4
  6. 2025Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decodingcited by 4
  7. 2026DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verificationcited by 4
  8. 2024Breaking the Attention Bottleneckcited by 1
  9. 2026Flow of Spans: Generalizing Language Models to Dynamic Span-Vocabulary via GFlowNetscited by 0
  10. 2026CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Accelerationcited by 0
  11. 2026Near-Exponential Convergence Rates for kNN Classification based on Boltzmann Margincited by 0
  12. 2026Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computescited by 0
  13. 2026Guaranteeing Faithful Evidence Extraction in Speculative Retrieval-Augmented Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  2. 2025Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies
  3. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  4. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding
  5. 2024Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters
  6. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement
  7. 2025READER: Retrieval-Assisted Drafter for Efficient LLM Inference
  8. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution