mlsanityData to 14 Oct 2026

Evaluating RAG Metrics in Applied Contexts: An Experiment, Its Findings and Its Limitations

arXiv 2607.07302 ↗First posted 8 Jul 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper reports an empirical study evaluating the relevance of several RAG metrics. The experiment is based on a question-answering dataset created by human annotators from business data. The generated responses and retrieved spans of a RAG system are scored using evaluation metrics from four libraries (Ragas, DeepEval, RAGChecker, Opik). These metrics are compared to scores given by two evaluators, as well as to standard metrics such as recall. An analysis of correlations is conducted. Finally, we highlight certain limitations of our methodology, compare it to those used in the literature, and suggest some avenues for future research. This paper is an English translation of a paper originally published in the French-speaking workshop EvalLLM (Brabant, 2026).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 16 references that are in the map. First posted on arXiv 8 Jul 2026.

Builds on

13 of 16 references.

References · 13

  1. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  2. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  3. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 418
  4. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  5. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
  6. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 72
  7. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 66
  8. 2024RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generationcited by 61
  9. 2023X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspectscited by 28
  10. 2024Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!cited by 21
  11. 2024SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Gradingcited by 14
  12. 2024Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluationcited by 4
1 more
  1. 2025Dynamic Knowledge Integration for Evidence-Driven Counter-Argument Generation with Large Language Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluation of RAG Metrics for Question Answering in the Telecom Domain
  2. 2024Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems
  3. 2024Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective
  4. 2024InspectorRAGet: An Introspection Platform for RAG Evaluation
  5. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems
  6. 2025Knowledge-Graph Based RAG System Evaluation Framework
  7. 2025The Viability of Crowdsourcing for RAG Evaluation
  8. 2026Automating Multi-Hop RAG Evaluation via TRIAD: From Context Extraction to Validated Dataset Generation