# RAGXplain: From Explainable Evaluation to Actionable Guidance of RAG Pipelines

arXiv: https://arxiv.org/abs/2505.13538 · HTML: https://mlsanity.com/paper/2505.13538/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 18 May 2025.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 27 of 27)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [Benchmarking Retrieval-Augmented Generation for Medicine](https://mlsanity.com/paper/2402.13178/) (2024)
- [ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2311.09476/) (2023)
- [ASQA: Factoid Questions Meet Long-Form Answers](https://mlsanity.com/paper/2204.06092/) (2022)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research](https://mlsanity.com/paper/2405.13576/) (2024)
- [RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2407.11005/) (2024)
- [ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding](https://mlsanity.com/paper/2501.07861/) (2025)
- [INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedback](https://mlsanity.com/paper/2305.14282/) (2023)
- [RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation](https://mlsanity.com/paper/2412.11919/) (2024)
- [WikiPassageQA: A Benchmark Collection for Research on Non-factoid Answer Passage Retrieval](https://mlsanity.com/paper/1805.03797/) (2018)
- [CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity](https://mlsanity.com/paper/2410.12248/) (2024)
- [Optimizing Retrieval Strategies for Financial Question Answering Documents in Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2503.15191/) (2025)
- [Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems](https://mlsanity.com/paper/2412.10571/) (2024)
- [Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models](https://mlsanity.com/paper/2502.18817/) (2025)
- [Evaluating Quality of Answers for Retrieval-Augmented Generation: A Strong LLM Is All You Need](https://mlsanity.com/paper/2406.18064/) (2024)
- [GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answering](https://mlsanity.com/paper/2409.06595/) (2024)
- [Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice](https://mlsanity.com/paper/2409.07713/) (2024)
- [Search-Engine-augmented Dialogue Response Generation with Cheaply Supervised Query Production](https://mlsanity.com/paper/2302.09300/) (2023)
- [Retrieval Augmented Generation Based LLM Evaluation For Protocol State Machine Inference With Chain-of-Thought Reasoning](https://mlsanity.com/paper/2502.15727/) (2025)

## Built on by (most cited first, 4 of 4)

- [RAGExplorer: A Visual Analytics System for the Comparative Diagnosis of RAG Systems](https://mlsanity.com/paper/2601.12991/) (2026)
- [RAG-X: Systematic Diagnosis of Retrieval-Augmented Generation for Medical Question Answering](https://mlsanity.com/paper/2603.03541/) (2026)
- [Don't Offer What Can't Be Done: Deterministic Executability Gating for LLM Skill Selection at Scale](https://mlsanity.com/paper/2608.01050/) (2026)
- [RECTIFY: An Interactive Workbench for Post-Evaluation RAG Diagnosis, Repair, and Verification](https://mlsanity.com/paper/2609.16764/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems couple large language models with external knowledge, yet most evaluation methods report aggregate scores that reveal whether a pipeline underperforms but not where or why. We introduce RAGXplain, an evaluation framework that translates performance metrics into actionable guidance. RAGXplain structures evaluation around a 'Metric Diamond' connecting user input, retrieved context, generated answer, and (when available) ground truth via six diagnostic dimensions. It uses LLM reasoning to produce natural-language failure-mode explanations and prioritized interventions. Across five QA benchmarks, applying RAGXplain's recommendations in a single human-guided pass consistently improves RAG pipeline performance across multiple metrics. We release RAGXplain as open source to support reproducibility and community adoption.