mlsanityData to 7 Oct 2026

Predictive Prefetching for Retrieval-Augmented Generation

arXiv 2605.17989 ↗First posted 18 May 2026cs.CL cs.AIcited by 0

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) improves factual grounding in large language models but suffers from substantial latency due to synchronous retrieval. While recent work explores asynchronous retrieval, existing approaches rely on heuristic coordination between retrieval and generation and assume stable information demands during decoding that often break in complex, multi-domain settings. In this paper, we propose an advanced asynchronous retrieval framework that enables predictive prefetching aligned with evolving information needs. The framework explicitly predicts when retrieval should be triggered and what information should be retrieved using three components, a retrieval predictor, a context monitor, and a query generator, by exploiting semantic precursors in generation dynamics that emerge several tokens before uncertainty becomes critical. Experiments on multiple benchmarks demonstrate up to 43.5% end-to-end latency reduction and 62.4% improvement in time-to-first-token, while maintaining answer quality comparable to synchronous RAG baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 32 of its 50 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

32 of 50 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks20 Dec
  2. 2023Active Retrieval Augmented Generation11 May
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 28

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2017Billion-scale similarity search with GPUscited by 2,477
  9. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  10. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  11. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  12. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
16 more
  1. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  2. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  3. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  4. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  5. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
  6. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  7. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  8. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  9. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  10. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  11. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147
  12. 2024HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extractioncited by 92
  13. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 76
  14. 2024Repoformer: Selective Retrieval for Repository-Level Code Completioncited by 55
  15. 2022Adapting Pretrained Text-to-Text Models for Long Text Sequencescited by 21
  16. 2024Not All Contexts Are Equal: Teaching LLMs Credibility-aware Generationcited by 21

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Retrieval as Generation: A Unified Framework with Self-Triggered Information Planning
  2. 2025Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decoding
  3. 2026To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval
  4. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  5. 2025ImpRAG: Retrieval-Augmented Generation with Implicit Queries
  6. 2026A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation
  7. 2025Optimizing Retrieval-Augmented Generation: Analysis of Hyperparameter Impact on Performance and Efficiency
  8. 2024Accelerating Retrieval-Augmented Generation