mlsanityData to 14 Oct 2026

StratRAG: A Multi-Hop Retrieval Evaluation Dataset for Retrieval-Augmented Generation Systems

arXiv 2604.22757 ↗First posted 6 Mar 2026cs.IR cs.AIcited by 0

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

also in 1 more
Abstract (arXiv)

We introduce StratRAG, an open-source retrieval evaluation dataset for benchmarking Retrieval-Augmented Generation (RAG) systems on multi-hop reasoning tasks under realistic, noisy document-pool conditions. Derived from HotpotQA (distractor setting), StratRAG comprises 2,200 examples across three question types -- bridge, comparison, and yes-no -- each paired with a pool of 15 candidate documents containing exactly 2 gold documents and 13 topically related distractors. We benchmark three retrieval strategies -- BM25, dense retrieval (all-MiniLM-L6-v2), and hybrid fusion -- reporting Recall@k, MRR, and NDCG@5 on the validation set. Hybrid retrieval achieves the best overall performance (Recall@2 = 0.70, MRR = 0.93), yet bridge questions remain substantially harder (Recall@2 = 0.67), motivating future work on reinforcement-learning-based retrieval policies. StratRAG is publicly available at https://huggingface.co/datasets/Aryanp088/StratRAG.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 9 of its 13 references that are in the map. First posted on arXiv 6 Mar 2026.

Builds on

9 of 13 references.

References · 9

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  6. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  7. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  8. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  9. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 418

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Hierarchical Retrieval-Augmented Generation Model with Rethink for Multi-hop Question Answering
  2. 2025A Comprehensive Evaluation of Transformer-Based Question Answering Models and RAG-Enhanced Design
  3. 2025Investigating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queries
  4. 2026A Reproducibility Study of Metacognitive Retrieval-Augmented Generation
  5. 2025Optimizing Multi-Hop Document Retrieval Through Intermediate Representations
  6. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  7. 2026Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategies
  8. 2026CIRAG: Construction-Integration Retrieval and Adaptive Generation for Multi-hop Question Answering