mlsanityData to 7 Oct 2026

LiveRAG: A diverse Q&A dataset with varying difficulty level for RAG evaluation

arXiv 2511.14531 ↗First posted 18 Nov 2025cs.CL cs.IRcited by 2

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

With Retrieval Augmented Generation (RAG) becoming more and more prominent in generative AI solutions, there is an emerging need for systematically evaluating their effectiveness. We introduce the LiveRAG benchmark, a publicly available dataset of 895 synthetic questions and answers designed to support systematic evaluation of RAG-based Q&A systems. This synthetic benchmark is derived from the one used during the SIGIR'2025 LiveRAG Challenge, where competitors were evaluated under strict time constraints. It is augmented with information that was not made available to competitors during the Challenge, such as the ground-truth answers, together with their associated supporting claims which were used for evaluating competitors' answers. In addition, each question is associated with estimated difficulty and discriminability scores, derived from applying an Item Response Theory model to competitors' responses. Our analysis highlights the benchmark's questions diversity, the wide range of their difficulty levels, and their usefulness in differentiating between system capabilities. The LiveRAG benchmark will hopefully help the community advance RAG research, conduct systematic evaluation, and develop more robust Q&A systems.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 22 of its 37 references that are in the map. First posted on arXiv 18 Nov 2025.

Builds on

22 of 37 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges21 Apr
  2. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems25 Jun
  3. 2024CRAG -- Comprehensive RAG Benchmark7 Jun
  4. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation26 Sep

Other references · 18

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2024A Survey on LLM-as-a-Judgecited by 1,384
  5. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  6. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  7. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  8. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  9. 2019ELI5: Long Form Question Answeringcited by 567
  10. 2021Curriculum Learning: A Surveycited by 257
  11. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  12. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
6 more
  1. 2019Synthetic QA Corpora Generation with Roundtrip Consistencycited by 143
  2. 2019Unsupervised Question Answering by Cloze Translationcited by 59
  3. 2021Comparing Test Sets with Item Response Theorycited by 55
  4. 2022py-irt: A Scalable Item Response Theory Library for Pythoncited by 19
  5. 2022What Makes Reading Comprehension Questions Difficult?cited by 11
  6. 2014On Big Data Benchmarkingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026ASTRA-QA: A Benchmark for Abstract Question Answering over Documentscited by 0
  2. 2026Telco-GAIA: Bilingual Benchmark for Agents in Telecom Domaincited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition
  2. 2025Generating Diverse Q&A Benchmarks for RAG Evaluation with DataMorgana
  3. 2025T2-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation
  4. 2025Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge
  5. 2025PreQRAG -- Classify and Rewrite for Enhanced RAG
  6. 2024RAGProbe: An Automated Approach for Evaluating RAG Applications
  7. 2024Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage
  8. 2025RMIT-ADM+S at the SIGIR 2025 LiveRAG Challenge