# GRADE: Generating multi-hop QA and fine-gRAined Difficulty matrix for RAG Evaluation

arXiv: https://arxiv.org/abs/2508.16994 · HTML: https://mlsanity.com/paper/2508.16994/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge-graph and structured-knowledge RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 23 Aug 2025.

## Placed in

- Retrieval-augmented generation / [Knowledge-graph and structured-knowledge RAG](https://mlsanity.com/rag/knowledge-graph-and-structured-knowledge-rag/)

## Builds on (most cited first, 20 of 20)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries](https://mlsanity.com/paper/2401.15391/) (2024)
- [Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.12941/) (2024)
- [Seven Failure Points When Engineering a Retrieval Augmented Generation System](https://mlsanity.com/paper/2401.05856/) (2024)
- [Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension](https://mlsanity.com/paper/2002.00293/) (2020)
- [Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach](https://mlsanity.com/paper/2407.16833/) (2024)
- [Compositional Questions Do Not Necessitate Multi-hop Reasoning](https://mlsanity.com/paper/1906.02900/) (2019)
- [Question Answering by Reasoning Across Documents with Graph Convolutional Networks](https://mlsanity.com/paper/1808.09920/) (2018)
- [A Reasoning-Focused Legal Retrieval Benchmark](https://mlsanity.com/paper/2505.03970/) (2025)
- [Answering Complex Questions by Joining Multi-Document Evidence with Quasi Knowledge Graphs](https://mlsanity.com/paper/1908.00469/) (2019)
- [Do Question Answering Modeling Improvements Hold Across Benchmarks?](https://mlsanity.com/paper/2102.01065/) (2021)
- [REAPER: Reasoning based Retrieval Planning for Complex RAG Systems](https://mlsanity.com/paper/2407.18553/) (2024)
- [Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems](https://mlsanity.com/paper/2411.19710/) (2024)
- [HOP, UNION, GENERATE: Explainable Multi-hop Reasoning without Rationale Supervision](https://mlsanity.com/paper/2305.14237/) (2023)
- [Performance Prediction for Multi-hop Questions](https://mlsanity.com/paper/2308.06431/) (2023)
- [MHTS: Multi-Hop Tree Structure Framework for Generating Difficulty-Controllable QA Datasets for RAG Evaluation](https://mlsanity.com/paper/2504.08756/) (2025)
- [Evaluating Multi-Hop Reasoning in Large Language Models: A Chemistry-Centric Case Study](https://mlsanity.com/paper/2504.16414/) (2025)

## Built on by (most cited first, 8 of 8)

- [MaxShapley: Towards Incentive-compatible Generative Search with Fair Context Attribution](https://mlsanity.com/paper/2512.05958/) (2025)
- [DocArena: Turning Raw Documents into Controllable Training Environments for Document Search Agents](https://mlsanity.com/paper/2606.26122/) (2026)
- [When Iterative RAG Beats Ideal Evidence: A Diagnostic Study in Scientific Multi-hop Question Answering](https://mlsanity.com/paper/2601.19827/) (2026)
- [A Fixed-Budget, Cluster-Aware Standard for LLM-as-a-Judge Evaluation: A Multi-Hop RAG Stress Test](https://mlsanity.com/paper/2605.27789/) (2026)
- [A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation](https://mlsanity.com/paper/2607.04784/) (2026)
- [LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis](https://mlsanity.com/paper/2607.06160/) (2026)
- [Automating Multi-Hop RAG Evaluation via TRIAD: From Context Extraction to Validated Dataset Generation](https://mlsanity.com/paper/2608.21558/) (2026)
- [AgentTune: A Unified Training Lifecycle for Tool-Using LLM Agents](https://mlsanity.com/paper/2610.13335/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems are widely adopted in knowledge-intensive NLP tasks, but current evaluations often overlook the structural complexity and multi-step reasoning required in real-world scenarios. These benchmarks overlook key factors such as the interaction between retrieval difficulty and reasoning depth. To address this gap, we propose GRADE, a novel evaluation framework that models task difficulty along two orthogonal dimensions: (1) reasoning depth, defined by the number of inference steps (hops), and (2) semantic distance between the query and its supporting evidence. We construct a synthetic multi-hop QA dataset from factual news articles by extracting knowledge graphs and augmenting them through semantic clustering to recover missing links, allowing us to generate diverse and difficulty-controlled queries. Central to our framework is a 2D difficulty matrix that combines generator-side and retriever-side difficulty. Experiments across multiple domains and models show that error rates strongly correlate with our difficulty measures, validating their diagnostic utility. GRADE enables fine-grained analysis of RAG performance and provides a scalable foundation for evaluating and improving multi-hop reasoning in real-world applications.