# IIRC: A Dataset of Incomplete Information Reading Comprehension Questions

arXiv: https://arxiv.org/abs/2011.07127 · HTML: https://mlsanity.com/paper/2011.07127/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2020.

## Built on by (most cited first, 30 of 53)

- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023)
- [Knowledge Graph Prompting for Multi-Document Question Answering](https://mlsanity.com/paper/2308.11730/) (2023)
- [Benchmark Data Contamination of Large Language Models: A Survey](https://mlsanity.com/paper/2406.04244/) (2024)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [Answering Questions by Meta-Reasoning over Multiple Chains of Thought](https://mlsanity.com/paper/2304.13007/) (2023)
- [DRAGIN: Dynamic Retrieval Augmented Generation based on the Information Needs of Large Language Models](https://mlsanity.com/paper/2403.10081/) (2024)
- [A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges](https://mlsanity.com/paper/2508.05668/) (2025)
- [The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models](https://mlsanity.com/paper/2310.11877/) (2023)
- [Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs](https://mlsanity.com/paper/2507.09477/) (2025)
- [SeaKR: Self-aware Knowledge Retrieval for Adaptive Retrieval Augmented Generation](https://mlsanity.com/paper/2406.19215/) (2024)
- [Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills](https://mlsanity.com/paper/2107.07261/) (2021)
- [Agent-Oriented Planning in Multi-Agent Systems](https://mlsanity.com/paper/2410.02189/) (2024)
- [End-to-End Beam Retrieval for Multi-Hop Question Answering](https://mlsanity.com/paper/2308.08973/) (2023)
- [ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers](https://mlsanity.com/paper/2110.06884/) (2021)
- [TC-RAG:Turing-Complete RAG's Case study on Medical LLM Systems](https://mlsanity.com/paper/2408.09199/) (2024)
- [English Machine Reading Comprehension Datasets: A Survey](https://mlsanity.com/paper/2101.10421/) (2021)
- [SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction](https://mlsanity.com/paper/2503.01478/) (2025)
- [Visconde: Multi-document QA with GPT-3 and Neural Reranking](https://mlsanity.com/paper/2212.09656/) (2022)
- [Modeling Multi-hop Question Answering as Single Sequence Prediction](https://mlsanity.com/paper/2205.09226/) (2022)
- [Husky: A Unified, Open-Source Language Agent for Multi-Step Reasoning](https://mlsanity.com/paper/2406.06469/) (2024)
- [Exploiting Reasoning Chains for Multi-hop Science Question Answering](https://mlsanity.com/paper/2109.02905/) (2021)
- [Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition](https://mlsanity.com/paper/2107.13935/) (2021)
- [Empowering Language Models with Active Inquiry for Deeper Understanding](https://mlsanity.com/paper/2402.03719/) (2024)
- [Mitigating False-Negative Contexts in Multi-document Question Answering with Retrieval Marginalization](https://mlsanity.com/paper/2103.12235/) (2021)
- [Graph Neural Network Enhanced Retrieval for Question Answering of LLMs](https://mlsanity.com/paper/2406.06572/) (2024)
- [Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering](https://mlsanity.com/paper/2311.13565/) (2023)
- [Modeling Uncertainty Trends for Timely Retrieval in Dynamic RAG](https://mlsanity.com/paper/2511.09980/) (2025)

## Abstract (arXiv)

Humans often have to read multiple documents to address their information needs. However, most existing reading comprehension (RC) tasks only focus on questions for which the contexts provide all the information required to answer them, thus not evaluating a system's performance at identifying a potential lack of sufficient information and locating sources for that information. To fill this gap, we present a dataset, IIRC, with more than 13K questions over paragraphs from English Wikipedia that provide only partial information to answer them, with the missing information occurring in one or more linked documents. The questions were written by crowd workers who did not have access to any of the linked documents, leading to questions that have little lexical overlap with the contexts where the answers appear. This process also gave many questions without answers, and those that require discrete reasoning, increasing the difficulty of the task. We follow recent modeling work on various reading comprehension datasets to construct a baseline model for this dataset, finding that it achieves 31.1% F1 on this task, while estimated human performance is 88.4%. The dataset, code for the baseline system, and a leaderboard can be found at https://allennlp.org/iirc.