mlsanityData to 7 Oct 2026

In Defense of RAG in the Era of Long-Context Language Models

arXiv 2409.01666 ↗First posted 3 Sep 2024cs.CLcited by 52

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Overcoming the limited context limitations in early-generation LLMs, retrieval-augmented generation (RAG) has been a reliable solution for context-based answer generation in the past. Recently, the emergence of long-context LLMs allows the models to incorporate much longer text sequences, making RAG less attractive. Recent studies show that long-context LLMs significantly outperform RAG in long-context applications. Unlike the existing works favoring the long-context LLM over RAG, we argue that the extremely long context in LLMs suffers from a diminished focus on relevant information and leads to potential degradation in answer quality. This paper revisits the RAG in long-context answer generation. We propose an order-preserve retrieval-augmented generation (OP-RAG) mechanism, which significantly improves the performance of RAG for long-context question-answer applications. With OP-RAG, as the number of retrieved chunks increases, the answer quality initially rises, and then declines, forming an inverted U-shaped curve. There exist sweet points where OP-RAG could achieve higher answer quality with much less tokens than long-context LLM taking the whole context as input. Extensive experiments on public benchmark demonstrate the superiority of our OP-RAG.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 15 of its 19 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

15 of 19 references.

References · 15

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  11. 2020Efficient Transformers: A Surveycited by 784
  12. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
3 more
  1. 2023Augmented Language Models: a Surveycited by 398
  2. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  3. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 117

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 30; 2026: 12.

52 papers.

Sort:

By area:

  1. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
  2. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systemscited by 69
  3. 2025ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agentscited by 64
  4. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  5. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 35
  6. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
  7. 2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystackcited by 20
  8. 2025Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-kcited by 19
  9. 2025TARGET: Benchmarking Table Retrieval for Generative Taskscited by 17
  10. 2024KBLaM: Knowledge Base augmented Language Modelcited by 16
  11. 2025GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?cited by 16
  12. 2024Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Datacited by 11
38 more
  1. 2025Keyword search is all you need: Achieving RAG-Level Performance without vector databases using agentic tool usecited by 10
  2. 2024Assessing Episodic Memory in LLMs with Sequence Order Recall Taskscited by 9
  3. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Modelscited by 9
  4. 2025Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processingcited by 7
  5. 2025POISONCRAFT: Practical Poisoning of Retrieval-Augmented Generation for Large Language Modelscited by 6
  6. 2024L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding?cited by 5
  7. 2025Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMscited by 5
  8. 2025FinGEAR: Financial Mapping-Guided Enhanced Answer Retrievalcited by 5
  9. 2025Revisiting Long-context Modeling from Context Denoising Perspectivecited by 5
  10. 2025Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRscited by 4
  11. 2024SayComply: Grounding Field Robotic Tasks in Operational Compliance through Retrieval-Based Language Modelscited by 3
  12. 2025Scaling Multi-Document Event Summarization: Evaluating Compression vs. Full-Text Approachescited by 3
  13. 2025Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformerscited by 3
  14. 2025Q-RAG: Long Context Multi-step Retrieval via Value-based Embedder Trainingcited by 3
  15. 2024Enhancing Long-term RAG Chatbots with Psychological Models of Memory Importance and Forgettingcited by 2
  16. 2024100% Elimination of Hallucinations on RAGTruth for GPT-4 and GPT-3.5 Turbocited by 2
  17. 2024Mixture-of-PageRanks: Replacing Long-Context with Real-Time, Sparse GraphRAGcited by 2
  18. 2025Fine-Tuning Large Language Models and Evaluating Retrieval Methods for Improved Question Answering on Building Codescited by 2
  19. 2025MacRAG: Compress, Slice, and Scale-up for Multi-Scale Adaptive Context RAGcited by 2
  20. 2025CAPRAG: A Large Language Model Solution for Customer Service and Automatic Reporting using Vector and Graph Retrieval-Augmented Generationcited by 1
  21. 2025Estimating Optimal Context Length for Hybrid Retrieval-augmented Multi-document Summarizationcited by 1
  22. 2025JointRank: Rank Large Set with Single Passcited by 1
  23. 2025Knowledge Fusion via Bidirectional Information Aggregationcited by 1
  24. 2025RAGalyst: Automated Human-Aligned Agentic Evaluation for Domain-Specific RAGcited by 1
  25. 2026Exploring Fine-Tuning for In-Context Retrieval and Efficient KV-Caching in Long-Context Language Modelscited by 1
  26. 2026LLM-Oriented Information Retrieval: A Denoising-First Perspectivecited by 1
  27. 2026Lost in the Evidence? Reproducing Document Position and Context Size Effects in RAGcited by 1
  28. 2026MM-BizRAG: Rethinking Multimodal Retrieval-Augmented Generation for General Purpose Enterprise Q&Acited by 1
  29. 2024Recursive Abstractive Processing for Retrieval in Dynamic Datasetscited by 0
  30. 2025FB-RAG: Improving RAG with Forward and Backward Lookupcited by 0
  31. 2025Agri-Query: A Case Study on RAG vs. Long-Context LLMs for Cross-Lingual Technical Question Answeringcited by 0
  32. 2025Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answeringcited by 0
  33. 2026Structured Style-Rewrite with Chain-of-Thought Planning for Low-Resource Character Dialoguecited by 0
  34. 2026PRAG: End-to-End Privacy-Preserving Retrieval-Augmented Generationcited by 0
  35. 2026Efficient Retrieval-Augmented Generation via Token Co-occurrence Graphscited by 0
  36. 2026Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPEcited by 0
  37. 2026MegaMem: A Retrieval Solution for Ultra-Large Context Windowscited by 0
  38. 2026The Laws of Context Allocation: Causal Measurement and Closed-Loop Orchestration in Generative Searchcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systems
  2. 2024Introducing a new hyper-parameter for RAG: Context Window Utilization
  3. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
  4. 2024Long2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall
  5. 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering
  6. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  7. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  8. 2024Context Embeddings for Efficient Answer Generation in RAG