mlsanityData to 7 Oct 2026

Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets

arXiv 2504.20119 ↗First posted 28 Apr 2025cs.IR cs.AIcited by 7

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) has advanced significantly in recent years. The complexity of RAG systems, which involve multiple components-such as indexing, retrieval, and generation-along with numerous other parameters, poses substantial challenges for systematic evaluation and quality enhancement. Previous research highlights that evaluating RAG systems is essential for documenting advancements, comparing configurations, and identifying effective approaches for domain-specific applications. This study systematically reviews 63 academic articles to provide a comprehensive overview of state-of-the-art RAG evaluation methodologies, focusing on four key areas: datasets, retrievers, indexing and databases, and the generator component. We observe the feasibility of an automated evaluation approach for each component of a RAG system, leveraging an LLM capable of both generating evaluation datasets and conducting evaluations. In addition, we found that further practical research is essential to provide companies with clear guidance on the do's and don'ts of implementing and evaluating RAG systems. By synthesizing evaluation approaches for key RAG components and emphasizing the creation and adaptation of domain-specific datasets for benchmarking, we contribute to the advancement of systematic evaluation methods and the improvement of evaluation rigor for RAG systems. Furthermore, by examining the interplay between automated approaches leveraging LLMs and human judgment, we contribute to the ongoing discourse on balancing automation and human input, clarifying their respective contributions, limitations, and challenges in achieving robust and reliable evaluations.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 57 of its 68 references that are in the map. First posted on arXiv 28 Apr 2025.

Builds on

57 of 68 references.

Closest ancestors · 27

Same direction, newest first

  1. 2024Long2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall30 Oct
  2. 2024MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems17 Oct
  3. 2024CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity16 Oct
  4. 2024A Methodology for Evaluating RAG Systems: A Case Study On Configuration Dependency Validation11 Oct
  5. 2024Intrinsic Evaluation of RAG Systems for Deep-Logic Questions3 Oct
  6. 2024RAGProbe: An Automated Approach for Evaluating RAG Applications24 Sep
  7. 2024IRSC: A Zero-shot Evaluation Benchmark for Information Retrieval through Semantic Comprehension in Retrieval-Augmented Generation Scenarios24 Sep
  8. 2024VERA: Validation and Evaluation of Retrieval-Augmented Systems16 Aug
  9. 2024RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation15 Aug
  10. 2024RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework2 Aug
  11. 2024Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QA22 Jul
  12. 2024Towards Optimizing and Evaluating a Retrieval Augmented QA Chatbot using LLMs with Human in the Loop8 Jul
15 more
  1. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generation1 Jul
  2. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems25 Jun
  3. 2024UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis21 Jun
  4. 2024Evaluating the Efficacy of Open-Source LLMs in Enterprise-Specific RAG Systems: A Comparative Study of Performance and Scalability17 Jun
  5. 2024DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generation9 Jun
  6. 2024CRAG -- Comprehensive RAG Benchmark7 Jun
  7. 2024Evaluation of Retrieval-Augmented Generation: A Survey13 May
  8. 2024Evaluating Retrieval Quality in Retrieval-Augmented Generation21 Apr
  9. 2024Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup26 Feb
  10. 2024FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generation19 Feb
  11. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models30 Jan
  12. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries27 Jan
  13. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  14. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov
  15. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation26 Sep

Other references · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  6. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  7. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  8. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  9. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158
  10. 2024LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domaincited by 113
  11. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 85
  12. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agentcited by 65
18 more
  1. 2024Lynx: An Open Source Hallucination Evaluation Modelcited by 55
  2. 2024RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answeringcited by 43
  3. 2023ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attackscited by 28
  4. 2024Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation Systemscited by 26
  5. 2024Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generationcited by 25
  6. 2024RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialoguescited by 19
  7. 2024Enhancing Q&A Text Retrieval with Ranking Models: Benchmarking, fine-tuning and deploying Rerankers for RAGcited by 16
  8. 2024ClimRetrieve: A Benchmarking Dataset for Information Retrieval from Corporate Climate Disclosurescited by 12
  9. 2024Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval Augmented Generation Systemscited by 12
  10. 2024Evaluating the Retrieval Component in LLM-Based Question Answering Systemscited by 11
  11. 2024Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Frameworkcited by 11
  12. 2024Face4RAG: Factual Consistency Evaluation for Retrieval Augmented Generation in Chinesecited by 9
  13. 2024Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coveragecited by 8
  14. 2024CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generationcited by 8
  15. 2024Should We Fine-Tune or RAG? Evaluating Different Techniques to Adapt LLMs for Dialoguecited by 7
  16. 2024Evaluating Quality of Answers for Retrieval-Augmented Generation: A Strong LLM Is All You Needcited by 7
  17. 2024FaaF: Facts as a Function for the evaluation of generated textcited by 5
  18. 2024WeQA: A Benchmark for Retrieval Augmented Generation in Wind Energy Domaincited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluationcited by 8
  3. 2025Rewrite-to-Rank: Optimizing Ad Visibility via Retrieval-Aware Text Rewritingcited by 2
  4. 2025Can we Evaluate RAGs with Synthetic Data?cited by 2
  5. 2025Methodological Framework for Quantifying Semantic Test Coverage in RAG Systemscited by 0
  6. 2026RAG-DIVE: A Dynamic Approach for Multi-Turn Dialogue Evaluation in Retrieval-Augmented Generationcited by 0
  7. 2026Automating Multi-Hop RAG Evaluation via TRIAD: From Context Extraction to Validated Dataset Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024InspectorRAGet: An Introspection Platform for RAG Evaluation
  2. 2025OmniBench-RAG: A Multi-Domain Evaluation Platform for Retrieval-Augmented Generation Tools
  3. 2025Engineering the RAG Stack: A Comprehensive Review of the Architecture and Trust Frameworks for Retrieval-Augmented Generation Systems
  4. 2024Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems
  5. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
  6. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  7. 2024Creating a Taxonomy for Retrieval Augmented Generation Applications
  8. 2025A System for Comprehensive Assessment of RAG Frameworks