mlsanityData to 14 Oct 2026

Towards a rigorous evaluation of RAG systems: the challenge of due diligence

arXiv 2507.21753 ↗First posted 29 Jul 2025cs.AI stat.APcited by 1

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

The rise of generative AI, has driven significant advancements in high-risk sectors like healthcare and finance. The Retrieval-Augmented Generation (RAG) architecture, combining language models (LLMs) with search engines, is particularly notable for its ability to generate responses from document corpora. Despite its potential, the reliability of RAG systems in critical contexts remains a concern, with issues such as hallucinations persisting. This study evaluates a RAG system used in due diligence for an investment fund. We propose a robust evaluation protocol combining human annotations and LLM-Judge annotations to identify system failures, like hallucinations, off-topic, failed citations, and abstentions. Inspired by the Prediction Powered Inference (PPI) method, we achieve precise performance measurements with statistical guarantees. We provide a comprehensive dataset for further analysis. Our contributions aim to enhance the reliability and scalability of RAG systems evaluation protocols in industrial applications.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 15 of its 23 references that are in the map. First posted on arXiv 29 Jul 2025.

Builds on

15 of 23 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generation24 Feb
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  3. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation26 Sep

Other references · 12

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  3. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 176
  4. 2023PPI++: Efficient Prediction-Powered Inferencecited by 117
  5. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  6. 2024AutoEval Done Right: Using Synthetic Data for Model Evaluationcited by 57
  7. 2024Active Statistical Inferencecited by 50
  8. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 49
  9. 2023Chainpoll: A high efficacy method for LLM hallucination detectioncited by 39
  10. 2024AI Hallucinations: A Misnomer Worth Clarifyingcited by 34
  11. 2024Cost-Effective Hallucination Detection for LLMscited by 18
  12. 2023Validating ChatGPT Facts through RDF Knowledge Graphs and Sentence Similaritycited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Methodological Framework for Quantifying Semantic Test Coverage in RAG Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024VERA: Validation and Evaluation of Retrieval-Augmented Systems
  2. 2024OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain
  3. 2026Enhancing LLMs with Context-Specific Knowledge for Mitigating Misinformation in SMEs: A RAG-based Modeling and Analysis
  4. 2025Towards Lighter and Robust Evaluation for Retrieval Augmented Generation
  5. 2024InspectorRAGet: An Introspection Platform for RAG Evaluation
  6. 2026Deepchecks: Evaluating Retrieval-Augmented Generation (RAG)
  7. 2024Deploying Large Language Models With Retrieval Augmented Generation
  8. 2024ERATTA: Extreme RAG for Table To Answers with Large Language Models