# Long Context vs. RAG for LLMs: An Evaluation and Revisits

arXiv: https://arxiv.org/abs/2501.01880 · HTML: https://mlsanity.com/paper/2501.01880/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- First posted on arXiv 27 Dec 2024.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Built on by (most cited first, 26 of 26)

- [Context Length Alone Hurts LLM Performance Despite Perfect Retrieval](https://mlsanity.com/paper/2510.05381/) (2025)
- [The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs](https://mlsanity.com/paper/2504.17768/) (2025)
- [Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks](https://mlsanity.com/paper/2504.18838/) (2025)
- [Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-k](https://mlsanity.com/paper/2506.08479/) (2025)
- [U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystack](https://mlsanity.com/paper/2503.00353/) (2025)
- [GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?](https://mlsanity.com/paper/2502.05252/) (2025)
- [FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.15800/) (2025)
- [RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding](https://mlsanity.com/paper/2502.20330/) (2025)
- [EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agents](https://mlsanity.com/paper/2601.16690/) (2026)
- [Retrieval--Reasoning Processes for Multi-hop Question Answering: A Four-Axis Design Framework and Empirical Trends](https://mlsanity.com/paper/2601.00536/) (2026)
- [AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Science](https://mlsanity.com/paper/2506.13992/) (2025)
- [VimRAG: Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph](https://mlsanity.com/paper/2602.12735/) (2026)
- [T2-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.12071/) (2025)
- [Is Progressive Disclosure All You Need for Long-Context Agents?](https://mlsanity.com/paper/2607.17598/) (2026)
- [OkraLong: A Flexible Retrieval-Augmented Framework for Long-Text Query Processing](https://mlsanity.com/paper/2503.02603/) (2025)
- [Mesh RAG: Retrieval Augmentation for Autoregressive Mesh Generation](https://mlsanity.com/paper/2511.16807/) (2025)
- [Exploring Fine-Tuning for In-Context Retrieval and Efficient KV-Caching in Long-Context Language Models](https://mlsanity.com/paper/2601.18527/) (2026)
- [P2RAG: Efficient Privacy-Preserving RAG Service Supporting Arbitrary Top-k Retrieval](https://mlsanity.com/paper/2603.14778/) (2026)
- [The Attribution Blind Spot: Detecting When Language Models Rely on Memory Rather Than Retrieved Context](https://mlsanity.com/paper/2605.26778/) (2026)
- [Dynamic Long Short-Term Memory Based Memory Storage For Long Horizon LLM Interaction](https://mlsanity.com/paper/2507.03042/) (2025)
- [Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering](https://mlsanity.com/paper/2510.18691/) (2025)
- [Beyond the Needle's Illusion: Decoupled Evaluation of Evidence Access and Use under Semantic Interference at 326M-Token Scale](https://mlsanity.com/paper/2601.20276/) (2026)
- [Retrieval Is Not Enough: Why Organizational AI Needs Epistemic Infrastructure](https://mlsanity.com/paper/2604.11759/) (2026)
- [FD-RAG: Federated Dual-System Retrieval-Augmented Generation](https://mlsanity.com/paper/2605.27432/) (2026)
- [Effects of Transcript Compression on LLM-based Medical Misinformation Detection in Japanese YouTube Videos](https://mlsanity.com/paper/2609.30882/) (2026)
- [UNREAL: Unifying Retrieval and Long-Context with a Single Model](https://mlsanity.com/paper/2610.08463/) (2026)

## Abstract (arXiv)

Extending context windows (i.e., Long Context, LC) and using retrievers to selectively access relevant information (i.e., Retrieval-Augmented Generation, RAG) are the two main strategies to enable LLMs to incorporate extremely long external contexts. This paper revisits recent studies on this topic, highlighting their key insights and discrepancies. We then provide a more comprehensive evaluation by filtering out questions answerable without external context, identifying the most effective retrieval methods, and expanding the datasets. We show that LC generally outperforms RAG in question-answering benchmarks, especially for Wikipedia-based questions. Summarization-based retrieval performs comparably to LC, while chunk-based retrieval lags behind. However, RAG has advantages in dialogue-based and general question queries. These insights underscore the trade-offs between RAG and LC strategies, offering guidance for future optimization of LLMs with external knowledge sources. We also provide an in-depth discussion on this topic, highlighting the overlooked importance of context relevance in existing studies.