mlsanityData to 7 Oct 2026

Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

arXiv 2407.16833 ↗First posted 23 Jul 2024cs.CL cs.AI cs.LGcited by 117

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval Augmented Generation (RAG) has been a powerful tool for Large Language Models (LLMs) to efficiently process overly lengthy contexts. However, recent LLMs like Gemini-1.5 and GPT-4 show exceptional capabilities to understand long contexts directly. We conduct a comprehensive comparison between RAG and long-context (LC) LLMs, aiming to leverage the strengths of both. We benchmark RAG and LC across various public datasets using three latest LLMs. Results reveal that when resourced sufficiently, LC consistently outperforms RAG in terms of average performance. However, RAG's significantly lower cost remains a distinct advantage. Based on this observation, we propose Self-Route, a simple yet effective method that routes queries to RAG or LC based on model self-reflection. Self-Route significantly reduces the computation cost while maintaining a comparable performance to LC. Our findings provide a guideline for long-context applications of LLMs using RAG and LC.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 30 of its 42 references that are in the map. First posted on arXiv 23 Jul 2024.

Builds on

30 of 42 references.

References · 30

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  12. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
18 more
  1. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  2. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  3. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  4. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  5. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  6. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  7. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  9. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  10. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  11. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
  12. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  13. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  14. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  15. 2024RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generationcited by 144
  16. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 116
  17. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  18. 2023LLatrieval: LLM-Verified Retrieval for Verifiable Generationcited by 52

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 61; 2026: 41.

117 papers.

Sort:

By area:

  1. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  2. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  3. 2024In Defense of RAG in the Era of Long-Context Language Modelscited by 52
  4. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Taskscited by 38
  5. 2025Parameters vs. Context: Fine-Grained Control of Knowledge Reliance in Language Modelscited by 38
  6. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 38
  7. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directionscited by 30
  8. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
  9. 2024HybGRAG: Hybrid Retrieval-Augmented Generation on Textual and Relational Knowledge Basescited by 26
  10. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontierscited by 26
  11. 2024Long Context RAG Performance of Large Language Modelscited by 23
  12. 2024HashAttention: Semantic Sparsity for Faster Inferencecited by 23
38 more
  1. 2025From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generationcited by 20
  2. 2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystackcited by 20
  3. 2025Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-kcited by 19
  4. 2025Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic Retrieval-Augmented Generation for Industry Challengescited by 19
  5. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
  6. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Servingcited by 18
  7. 2025FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generationcited by 17
  8. 2025TARGET: Benchmarking Table Retrieval for Generative Taskscited by 17
  9. 2024KBLaM: Knowledge Base augmented Language Modelcited by 16
  10. 2024UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generationcited by 14
  11. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decodingcited by 14
  12. 2025TeleMem: Building Long-Term and Multimodal Memory for Agentic AIcited by 14
  13. 2025Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimationcited by 12
  14. 2024Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Datacited by 11
  15. 2025Syntriever: How to Train Your Retriever with Synthetic Data from LLMscited by 11
  16. 2026Coding Agents are Effective Long-Context Processorscited by 11
  17. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  18. 2024A Reality Check on Context Utilisation for Retrieval-Augmented Generationcited by 9
  19. 2025Putting It All into Context: Simplifying Agents with LCLMscited by 9
  20. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Modelscited by 9
  21. 2025CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoningcited by 9
  22. 2025MIRIAD: Augmenting LLMs with millions of medical query-response pairscited by 8
  23. 2025Text or Pixels? It Takes Half: On the Token Efficiency of Visual Text Inputs in Multimodal LLMscited by 8
  24. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  25. 2025GRADE: Generating multi-hop QA and fine-gRAined Difficulty matrix for RAG Evaluationcited by 7
  26. 2025A3: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Servingcited by 7
  27. 2026Mnemis: Dual-Route Retrieval on Hierarchical Graphs for Long-Term LLM Memorycited by 7
  28. 2025Single-Pass Document Scanning for Question Answeringcited by 6
  29. 2025vAttention: Verified Sparse Attentioncited by 6
  30. 2025VersionRAG: Version-Aware Retrieval-Augmented Generation for Evolving Documentscited by 6
  31. 2025Do LLMs Truly Understand When a Precedent Is Overruled?cited by 6
  32. 2024L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding?cited by 5
  33. 2025Overflow Prevention Enhances Long-Context Recurrent LLMscited by 5
  34. 2025LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation frameworkcited by 5
  35. 2025Safeguarding RAG Pipelines with GMTP: A Gradient-based Masked Token Probability Method for Poisoned Document Detectioncited by 5
  36. 2025A Theory of Adaptive Scaffolding for LLM-Based Pedagogical Agentscited by 5
  37. 2026MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systemscited by 5
  38. 2026RaMem: Contextual Reinstatement for Long-term Agentic Memorycited by 5

50 most cited of 117 All 117 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  3. 2025Tuning LLMs by RAG Principles: Towards LLM-native Memory
  4. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely
  5. 2025Does RAG Really Perform Bad For Long-Context Processing?
  6. 2024Contextual Compression in Retrieval-Augmented Generation for Large Language Models: A Survey
  7. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  8. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation