mlsanityData to 7 Oct 2026

Long Context RAG Performance of Large Language Models

arXiv 2411.03538 ↗First posted 5 Nov 2024cs.LG cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

Retrieval Augmented Generation (RAG) has emerged as a crucial technique for enhancing the accuracy of Large Language Models (LLMs) by incorporating external information. With the advent of LLMs that support increasingly longer context lengths, there is a growing interest in understanding how these models perform in RAG scenarios. Can these new long context models improve RAG performance? This paper presents a comprehensive study of the impact of increased context length on RAG performance across 20 popular open source and commercial LLMs. We ran RAG workflows while varying the total context length from 2,000 to 128,000 tokens (and 2 million tokens when possible) on three domain-specific datasets, and report key insights on the benefits and limitations of long context in RAG applications. Our findings reveal that while retrieving more documents can improve performance, only a handful of the most recent state of the art LLMs can maintain consistent accuracy at long context above 64k tokens. We also identify distinct failure modes in long context scenarios, suggesting areas for future research.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

25 of 33 references.

References · 25

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  9. 2024Mixtral of Expertscited by 1,785
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  12. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
13 more
  1. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  2. 2020Nearest Neighbor Machine Translationcited by 209
  3. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  4. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 117
  5. 2022Task-aware Retrieval with Instructionscited by 107
  6. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  7. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  8. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  9. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 53
  10. 2023On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codexcited by 41
  11. 2024Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLPcited by 27
  12. 2022Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answeringcited by 19
  13. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 15; 2026: 7.

23 papers.

Sort:

By area:

  1. 2025Scaling Long-Horizon LLM Agent via Context-Foldingcited by 136
  2. 2025Dynamic Parametric Retrieval Augmented Generation for Test-time Knowledge Enhancementcited by 24
  3. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16
  4. 2025TracLLM: A Generic Framework for Attributing Long Context LLMscited by 11
  5. 2025CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoningcited by 9
  6. 2026Symphony: A Cognitively-Inspired Multi-Agent System for Long-Video Understandingcited by 8
  7. 2025Position: Stop Acting Like Language Model Agents Are Normal Agentscited by 5
  8. 2025Ranking Free RAG: Replacing Re-ranking with Selection in RAG for Sensitive Domainscited by 5
  9. 2025Agent Identity Evals: Measuring Agentic Identitycited by 5
  10. 2025Optimizing Retrieval-Augmented Generation: Analysis of Hyperparameter Impact on Performance and Efficiencycited by 2
  11. 2025Model Hemorrhage and the Robustness Limits of Large Language Modelscited by 1
  12. 2026Replicating Belief, Not Bits: Epistemic State Replication for Agentic Systemscited by 1
11 more
  1. 2026Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Modelcited by 1
  2. 2025Stay Hungry, Stay Foolish: On the Extended Reading Articles Generation with LLMscited by 0
  3. 2025Enhancing Cache-Augmented Generation (CAG) with Adaptive Contextual Compression for Scalable Knowledge Integrationcited by 0
  4. 2025Augmenting LLM Reasoning with Dynamic Notes Writing for Complex QAcited by 0
  5. 2025Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Groundingcited by 0
  6. 2025Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answeringcited by 0
  7. 2026Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMscited by 0
  8. 2025Quantifying Document Impact in RAG-LLMscited by 0
  9. 2026Retrieval Is Not Enough: Why Organizational AI Needs Epistemic Infrastructurecited by 0
  10. 2026MimirRAG: A Multi-Agent RAG Framework for Financial Data Retrieval with Metadata Integrationcited by 0
  11. 2026Neurosymbolics for Data Engineering: Achieving Long Context Token Reduction Without Finetuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models
  2. 2025More Documents, Same Length: Isolating the Challenge of Multiple Documents in RAG
  3. 2025Does RAG Really Perform Bad For Long-Context Processing?
  4. 2024Contextual Compression in Retrieval-Augmented Generation for Large Language Models: A Survey
  5. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  6. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routing
  7. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
  8. 2024Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference