IRB: Automated Generation of Robust Factuality Benchmarks
On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval
Abstract (arXiv)
Static benchmarks for RAG systems often suffer from rapid saturation and require significant manual effort to maintain robustness. To address this, we present IRB, a framework for automatically generating benchmarks to evaluate the factuality of RAG systems. IRB employs a structured generation pipeline utilizing textitfactual scaffold and textitalgorithmic scaffold. We utilize IRB to construct a benchmark and evaluate frontier LLMs and retrievers. Our results demonstrate that IRB poses a significant challenge for frontier LLMs in the closed-book setting. Furthermore, our evaluation suggests that reasoning LLMs are more reliable, and that improving the retrieval component may yield more cost-effective gains in RAG system correctness than scaling the generator.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 20 of its 67 references that are in the map. First posted on arXiv 8 Feb 2026.
Builds on
20 of 67 references.
Closest ancestors · 1
Same direction, newest first
Other references · 19
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
- 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
- 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
- 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
- 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
- 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
- 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
- 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
7 more
- 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 283
- 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 131
- 2025SealQA: Raising the Bar for Reasoning in Search-Augmented Language Modelscited by 75
- 2024Molecular Facts: Desiderata for Decontextualization in LLM Fact Verificationcited by 44
- 2021Guiding the Growth: Difficulty-Controllable Question Generation through Step-by-Step Rewritingcited by 30
- 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
- 2025CiteGuard: Faithful Citation Attribution for LLMs via Retrieval-Augmented Validationcited by 9
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases
- 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation
- 2026Is Conformal Factuality for RAG-based LLMs Robust? Novel Metrics and Systematic Insights
- 2024AIC CTU system at AVeriTeC: Re-framing automated fact-checking as a simple RAG task
- 2026Beyond Factual Accuracy: Evaluating Global Reasoning Integrity in RAG Systems with LogicScore
- 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs
- 2026Diagnosing and Repairing Factual Errors in RAG under Budget Constraints
- 2025Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievals