mlsanityData to 14 Oct 2026

Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting

arXiv 2407.08223 ↗First posted 11 Jul 2024cs.CL cs.AIcited by 77

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval augmented generation (RAG) combines the generative abilities of large language models (LLMs) with external knowledge sources to provide more accurate and up-to-date responses. Recent RAG advancements focus on improving retrieval outcomes through iterative LLM refinement or self-critique capabilities acquired through additional instruction tuning of LLMs. In this work, we introduce Speculative RAG - a framework that leverages a larger generalist LM to efficiently verify multiple RAG drafts produced in parallel by a smaller, distilled specialist LM. Each draft is generated from a distinct subset of retrieved documents, offering diverse perspectives on the evidence while reducing input token counts per draft. This approach enhances comprehension of each subset and mitigates potential position bias over long context. Our method accelerates RAG by delegating drafting to the smaller specialist LM, with the larger generalist LM performing a single verification pass over the drafts. Extensive experiments demonstrate that Speculative RAG achieves state-of-the-art performance with reduced latency on TriviaQA, MuSiQue, PopQA, PubHealth, and ARC-Challenge benchmarks. It notably enhances accuracy by up to 12.97% while reducing latency by 50.83% compared to conventional RAG systems on PubHealth.

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 64 of its 70 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

64 of 70 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Corrective Retrieval Augmented Generation29 Jan
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec

Other references · 62

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  9. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  11. 2023Mistral 7Bcited by 3,164
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
38 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  4. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  5. 2024Mixtral of Expertscited by 1,823
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  7. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  8. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  9. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  10. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  11. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  12. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  13. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  14. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  15. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  16. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  17. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  18. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  19. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  20. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  21. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 532
  22. 2023Active Retrieval Augmented Generationcited by 516
  23. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  24. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  25. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  26. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 332
  27. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  28. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  29. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 269
  30. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 256
  31. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  32. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 213
  33. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  34. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 184
  35. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 171
  36. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 111
  37. 2023Dense X Retrieval: What Retrieval Granularity Should We Use?cited by 88
  38. 2023Learning to Filter Context for Retrieval-Augmented Generationcited by 83

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 37; 2026: 27.

77 papers.

Sort:

By area:

  1. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systemscited by 72
  2. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 63
  3. 2024Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Modelscited by 58
  4. 2024Rationale-Guided Retrieval Augmented Generation for Medical Question Answeringcited by 41
  5. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 36
  6. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 35
  7. 2025Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptancecited by 32
  8. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  9. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 23
  10. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 22
  11. 2025LeanRAG: Knowledge-Graph-Based Generation with Semantic Aggregation and Hierarchical Retrievalcited by 16
  12. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decodingcited by 14
38 more
  1. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 14
  2. 2025CausalRAG: Integrating Causal Graphs into Retrieval-Augmented Generationcited by 12
  3. 2024A Collaborative Multi-Agent Approach to Retrieval-Augmented Generation Across Diverse Datacited by 10
  4. 2025Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verificationcited by 10
  5. 2025Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systemscited by 10
  6. 2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrievalcited by 9
  7. 2025RAGRouter: Learning to Route Queries to Multiple Retrieval-Augmented Language Modelscited by 8
  8. 2025Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlapscited by 6
  9. 2025Hybrid AI for Responsive Multi-Turn Online Conversations with Novel Dynamic Routing and Feedback Adaptationcited by 6
  10. 2025Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decodingcited by 5
  11. 2025Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimizationcited by 5
  12. 2025Curriculum Guided Reinforcement Learning for Efficient Multi Hop Retrieval Augmented Generationcited by 5
  13. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  14. 2025OpenRAG: Optimizing RAG End-to-End via In-Context Retrieval Learningcited by 4
  15. 2025After Retrieval, Before Generation: Enhancing the Trustworthiness of Large Language Models in Retrieval-Augmented Generationcited by 4
  16. 2025SemRAG: Semantic Knowledge-Augmented RAG for Improved Question-Answeringcited by 4
  17. 2026Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMscited by 4
  18. 2026PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Servingcited by 4
  19. 2025From Virtual Agents to Robot Teams: A Multi-Robot Framework Evaluation in High-Stakes Healthcare Contextcited by 3
  20. 2026CLEAR: Cross-Lingual Enhancement in Alignment via Reverse-trainingcited by 3
  21. 2026RCBSF: A Multi-Agent Framework for Automated Contract Revision via Stackelberg Gamecited by 3
  22. 2025URAG: Implementing a Unified Hybrid RAG for Precise Answers in University Admission Chatbots -- A Case Study at HCMUTcited by 2
  23. 2025RAGDoll: Efficient Offloading-based Online RAG System on a Single GPUcited by 2
  24. 2025WebRec: Enhancing LLM-based Recommendations with Attention-guided RAG from Webcited by 2
  25. 2025Test-time Scaling of LLMs: A Survey from A Subproblem Structure Perspectivecited by 2
  26. 2026HCRE: LLM-based Hierarchical Classification for Cross-Document Relation Extraction with a Prediction-then-Verification Strategycited by 2
  27. 2026SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agentscited by 2
  28. 2024A Survey of Query Optimization in Large Language Modelscited by 1
  29. 2024Contrato360 2.0: A Document and Database-Driven Question-Answer System using Large Language Models and Agentscited by 1
  30. 2025Tutorial Proposal: Speculative Decoding for Efficient LLM Inferencecited by 1
  31. 2025Leveraging Approximate Caching for Faster Retrieval-Augmented Generationcited by 1
  32. 2025Instruction-Oriented Preference Alignment for Enhancing Multi-Modal Comprehension Capability of MLLMscited by 1
  33. 2025Question-Aware Knowledge Graph Prompting for Enhancing Large Language Modelscited by 1
  34. 2025Noise Augmented Fine Tuning for Mitigating Hallucinations in Large Language Modelscited by 1
  35. 2025Optimizing Retrieval for RAG via Reinforcement Learningcited by 1
  36. 2025Injecting Falsehoods: Adversarial Man-in-the-Middle Attacks Undermining Factual Recall in LLMscited by 1
  37. 2026FastV-RAG: Towards Fast and Fine-Grained Video QA with Retrieval-Augmented Generationcited by 1
  38. 2026PRISMA: Reinforcement Learning Guided Two-Stage Policy Optimization in Multi-Agent Architecture for Open-Domain Multi-Hop Question Answeringcited by 1

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
  2. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
  3. 2024RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation
  4. 2024The Power of Noise: Redefining Retrieval for RAG Systems
  5. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  6. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  7. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Models
  8. 2024Accelerating Retrieval-Augmented Generation