mlsanityData to 14 Oct 2026

LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routing

arXiv 2502.09977 ↗First posted 14 Feb 2025cs.CL cs.AIcited by 29

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Effectively incorporating external knowledge into Large Language Models (LLMs) is crucial for enhancing their capabilities and addressing real-world needs. Retrieval-Augmented Generation (RAG) offers an effective method for achieving this by retrieving the most relevant fragments into LLMs. However, the advancements in context window size for LLMs offer an alternative approach, raising the question of whether RAG remains necessary for effectively handling external knowledge. Several existing studies provide inconclusive comparisons between RAG and long-context (LC) LLMs, largely due to limitations in the benchmark designs. In this paper, we present LaRA, a novel benchmark specifically designed to rigorously compare RAG and LC LLMs. LaRA encompasses 2326 test cases across four practical QA task categories and three types of naturally occurring long texts. Through systematic evaluation of seven open-source and four proprietary LLMs, we find that the optimal choice between RAG and LC depends on a complex interplay of factors, including the model's parameter size, long-text capabilities, context length, task type, and the characteristics of the retrieved chunks. Our findings provide actionable guidelines for practitioners to effectively leverage both RAG and LC approaches in developing and deploying LLM applications. Our code and dataset is provided at: hrefhttps://github.com/Alibaba-NLP/LaRAhttps://github.com/Alibaba-NLP/LaRA.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 36 of its 43 references that are in the map. First posted on arXiv 14 Feb 2025.

Builds on

36 of 43 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Long2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall30 Oct
  2. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach23 Jul
  3. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models30 Jan
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  5. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov

Other references · 31

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  11. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  12. 2017The NarrativeQA Reading Comprehension Challengecited by 597
19 more
  1. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  2. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  3. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  4. 2018A Survey on Deep Learning for Named Entity Recognitioncited by 282
  5. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  6. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  7. 2017Why We Need New Evaluation Metrics for NLGcited by 233
  8. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  9. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  10. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  11. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  12. 2023Retrieval meets Long Context Large Language Modelscited by 115
  13. 2021Improving Named Entity Recognition by External Context Retrieving and Cooperative Learningcited by 76
  14. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 56
  15. 2023Calibrating LLM-Based Evaluatorcited by 53
  16. 2024In Defense of RAG in the Era of Long-Context Language Modelscited by 53
  17. 2022DAMO-NLP at SemEval-2022 Task 11: A Knowledge-based System for Multilingual Named Entity Recognitioncited by 17
  18. 2024RetrievalQA: Assessing Adaptive Retrieval-Augmented Generation for Short-form Open-Domain Question Answeringcited by 13
  19. 2024Groundedness in Retrieval-augmented Long-form Generation: An Empirical Studycited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 10.

29 papers.

Sort:

By area:

  1. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  2. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 229
  3. 2025Tongyi DeepResearch Technical Reportcited by 66
  4. 2025WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learningcited by 61
  5. 2025Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMscited by 12
  6. 2025RAGRouter: Learning to Route Queries to Multiple Retrieval-Augmented Language Modelscited by 8
  7. 2025Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Findcited by 7
  8. 2025PRELUDE: A Benchmark Designed to Require Global Comprehension and Reasoning over Long Contextscited by 7
  9. 2025Parametric Retrieval-Augmented Generation using Latent Routing of LoRA Adapterscited by 6
  10. 2026RAGRouter-Bench: A Dataset and Benchmark for Adaptive RAG Routingcited by 6
  11. 2025Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRscited by 4
  12. 2025PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimizationcited by 3
17 more
  1. 2026URAG: A Benchmark for Uncertainty Quantification in Retrieval-Augmented Large Language Modelscited by 3
  2. 2026How Retrieved Context Shapes Internal Representations in RAGcited by 2
  3. 2025OmniBench-RAG: A Multi-Domain Evaluation Platform for Retrieval-Augmented Generation Toolscited by 1
  4. 2025Incentive-Aligned Multi-Source LLM Summariescited by 1
  5. 2026NC-Bench: An LLM Benchmark for Evaluating Conversational Competencecited by 1
  6. 2026S3-Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inferencecited by 1
  7. 2026R3AG: Retriever Routing for Retrieval-Augmented Generationcited by 1
  8. 2025Context Is Not Comprehensioncited by 0
  9. 2025Agri-Query: A Case Study on RAG vs. Long-Context LLMs for Cross-Lingual Technical Question Answeringcited by 0
  10. 2025Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Groundingcited by 0
  11. 2025On the Sensitivity of Instruction-tuned LLMs to Harmful Sentences in Long Inputscited by 0
  12. 2025Flash-Fusion: Enabling Expressive, Low-Latency Queries on IoT Sensor Streams with LLMscited by 0
  13. 2025SagaScale: A Realistic, Scalable, and High-Quality Long-Context Benchmark Built from Full-Length Novelscited by 0
  14. 2026Human-Like Lifelong Memory: A Neuroscience-Grounded Architecture for Infinite Interactioncited by 0
  15. 2026Trace Only What You Need: Structure-Aware On-Demand Hypergraph Memory for Long-Document Question Answeringcited by 0
  16. 2026MGAL: A Multilingual Granularity-Aware Long-Context Benchmarkcited by 0
  17. 2026LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
  2. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models
  3. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  4. 2024Long Context RAG Performance of Large Language Models
  5. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely
  6. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks
  7. 2024Long Context vs. RAG for LLMs: An Evaluation and Revisits
  8. 2025On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systems