mlsanityData to 14 Oct 2026

Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey

arXiv 2504.14891 ↗First posted 21 Apr 2025cs.CLcited by 20

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Recent advancements in Retrieval-Augmented Generation (RAG) have revolutionized natural language processing by integrating Large Language Models (LLMs) with external information retrieval, enabling accurate, up-to-date, and verifiable text generation across diverse applications. However, evaluating RAG systems presents unique challenges due to their hybrid architecture that combines retrieval and generation components, as well as their dependence on dynamic knowledge sources in the LLM era. In response, this paper provides a comprehensive survey of RAG evaluation methods and frameworks, systematically reviewing traditional and emerging evaluation approaches, for system performance, factual accuracy, safety, and computational efficiency in the LLM era. We also compile and categorize the RAG-specific datasets and evaluation frameworks, conducting a meta-analysis of evaluation practices in high-impact RAG research. To the best of our knowledge, this work represents the most comprehensive survey for RAG evaluation, bridging traditional and LLM-driven methods, and serves as a critical resource for advancing RAG development.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 101 of its 124 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

101 of 124 references.

Closest ancestors · 26

Same direction, newest first

  1. 2025A System for Comprehensive Assessment of RAG Frameworks10 Apr
  2. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generation11 Mar
  3. 2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystack1 Mar
  4. 2025Fast or Better? Balancing Accuracy and Cost in Retrieval-Augmented Generation with Flexible User Control17 Feb
  5. 2024Long2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall30 Oct
  6. 2024CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity16 Oct
  7. 2024Toward General Instruction-Following Alignment for Retrieval-Augmented Generation12 Oct
  8. 2024VERA: Validation and Evaluation of Retrieval-Augmented Systems16 Aug
  9. 2024RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework2 Aug
  10. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems25 Jun
  11. 2024On the Role of Long-tail Knowledge in Retrieval Augmented Large Language Models24 Jun
  12. 2024DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generation9 Jun
14 more
  1. 2024CRAG -- Comprehensive RAG Benchmark7 Jun
  2. 2024Evaluation of Retrieval-Augmented Generation: A Survey13 May
  3. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models10 May
  4. 2024Evaluating Retrieval Quality in Retrieval-Augmented Generation21 Apr
  5. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Models17 Apr
  6. 2024Retrieval-Augmented Generation for AI-Generated Content: A Survey29 Feb
  7. 2024FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generation19 Feb
  8. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models30 Jan
  9. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries27 Jan
  10. 2024The Chronicles of RAG: The Retriever, the Chunk and the Generator15 Jan
  11. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  12. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov
  13. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation26 Sep
  14. 2022A Survey on Retrieval-Augmented Text Generation2 Feb

Other references · 75

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2023A Survey of Large Language Modelscited by 2,576
  8. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  9. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  10. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  11. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  12. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
38 more
  1. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  2. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  3. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  4. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  5. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  6. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  7. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
  8. 2024The Faiss librarycited by 376
  9. 2023GPTScore: Evaluate as You Desirecited by 361
  10. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
  11. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  12. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  13. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 267
  14. 2022RealTime QA: What's the Answer Right Now?cited by 244
  15. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  16. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  17. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 200
  18. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  19. 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
  20. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)cited by 159
  21. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 125
  22. 2025MMTEB: Massive Multilingual Text Embedding Benchmarkcited by 125
  23. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  24. 2024LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domaincited by 118
  25. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 117
  26. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 102
  27. 2024ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretabilitycited by 101
  28. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  29. 2023LLM-Rec: Personalized Recommendation via Prompting Large Language Modelscited by 77
  30. 2023Generative Recommendation: Towards Next-generation Recommender Paradigmcited by 70
  31. 2023RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledgecited by 51
  32. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systemscited by 51
  33. 2024Phantom: General Backdoor Attacks on Retrieval Augmented Language Generationcited by 49
  34. 2023From task structures to world models: What do LLMs know?cited by 46
  35. 2024Financial Report Chunking for Effective Retrieval Augmented Generationcited by 46
  36. 2024How Easily do Irrelevant Inputs Skew the Responses of Large Language Models?cited by 44
  37. 2024FairRAG: Fair Human Generation via Fair Retrieval Augmentationcited by 34
  38. 2019Conversing by Reading: Contentful Neural Conversation with On-demand Machine Readingcited by 32

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 14.

20 papers.

Sort:

By area:

  1. 2025Get Experience from Practice: LLM Agents with Record & Replaycited by 23
  2. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  3. 2026OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answeringcited by 5
  4. 2025LLM-Centric RAG with Multi-Granular Indexing and Confidence Constraintscited by 3
  5. 2025A Principle-based Framework for the Development and Evaluation of Large Language Models for Health and Wellnesscited by 3
  6. 2026Securing Retrieval-Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directionscited by 2
  7. 2026VikingMem: A Memory Base Management System for Stateful LLM-based Applicationscited by 2
  8. 2026Testing Retrieval-Augmented Generation Systems with Chunk Coveragecited by 2
  9. 2026When Knowledge Changes: Metamorphic Testing of RAG Systems with Mutationscited by 2
  10. 2025InterpDetect: Interpretable Signals for Detecting Hallucinations in Retrieval-Augmented Generationcited by 1
  11. 2026Empirical Evaluation of PDF Parsing and Chunking for Financial Question Answering with RAGcited by 1
  12. 2026Retrieval-Augmented Generation Must Move Beyond Factual Grounding to Represent Diverse Opinionscited by 1
8 more
  1. 2026Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiabilitycited by 1
  2. 2025RAGVUE: A Diagnostic View for Explainable and Automated Evaluation of Retrieval-Augmented Generationcited by 0
  3. 2025M'as contexto no es mejor. Paradoja de la diluci'on vectorial en RAG corporativoscited by 0
  4. 2026Guaranteeing Knowledge Integration with Joint Decoding for Retrieval-Augmented Generationcited by 0
  5. 2026Adaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power Meancited by 0
  6. 2026Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)cited by 0
  7. 2026Retrieval Augmented Generation Framework for the Nepali Legal Domain Question Answeringcited by 0
  8. 2026Retrieve, Reproduce, Reveal: Dissecting Retrieval-Augmented Software Vulnerability Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  2. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  3. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  4. 2024Retrieval-Augmented Generation for Natural Language Processing: A Survey
  5. 2025RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation
  6. 2025Dynamic and Parametric Retrieval-Augmented Generation
  7. 2024Deploying Large Language Models With Retrieval Augmented Generation
  8. 2025Enhancing Retrieval-Augmented Generation: A Study of Best Practices