# Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval

arXiv: https://arxiv.org/abs/2010.11915 · HTML: https://mlsanity.com/paper/2010.11915/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2020.

## Built on by (most cited first, 30 of 44)

- [PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them](https://mlsanity.com/paper/2102.07033/) (2021)
- [Fine-grained Hallucination Detection and Editing for Language Models](https://mlsanity.com/paper/2401.06855/) (2024)
- [Know Your Limits: A Survey of Abstention in Large Language Models](https://mlsanity.com/paper/2407.18418/) (2024)
- [The Art of Saying No: Contextual Noncompliance in Language Models](https://mlsanity.com/paper/2407.12043/) (2024)
- [DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering](https://mlsanity.com/paper/2211.05655/) (2022)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [Reliable, Adaptable, and Attributable Language Models with Retrieval](https://mlsanity.com/paper/2403.03187/) (2024)
- [CREPE: Open-Domain Question Answering with False Presuppositions](https://mlsanity.com/paper/2211.17257/) (2022)
- [The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models](https://mlsanity.com/paper/2310.11877/) (2023)
- [(QA)2: Question Answering with Questionable Assumptions](https://mlsanity.com/paper/2212.10003/) (2022)
- [Won't Get Fooled Again: Answering Questions with False Premises](https://mlsanity.com/paper/2307.02394/) (2023)
- [Can NLI Models Verify QA Systems' Predictions?](https://mlsanity.com/paper/2104.08731/) (2021)
- [One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval](https://mlsanity.com/paper/2107.11976/) (2021)
- [Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering](https://mlsanity.com/paper/2101.00391/) (2021)
- [Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2112.08688/) (2021)
- [Improving Transformers with Probabilistic Attention Keys](https://mlsanity.com/paper/2110.08678/) (2021)
- [Characterizing LLM Abstention Behavior in Science QA with Context Perturbations](https://mlsanity.com/paper/2404.12452/) (2024)
- [NoiseQA: Challenge Set Evaluation for User-Centric Question Answering](https://mlsanity.com/paper/2102.08345/) (2021)
- [Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answering](https://mlsanity.com/paper/2311.09542/) (2023)
- [Calibration of Machine Reading Systems at Scale](https://mlsanity.com/paper/2203.10623/) (2022)
- [MIA 2022 Shared Task: Evaluating Cross-lingual Open-Retrieval Question Answering for 16 Diverse Languages](https://mlsanity.com/paper/2207.00758/) (2022)
- [Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers](https://mlsanity.com/paper/2507.10787/) (2025)
- ["I Never Said That": A dataset, taxonomy and baselines on response clarity classification](https://mlsanity.com/paper/2409.13879/) (2024)
- [MultiHoax: A Dataset of Multi-hop False-Premise Questions](https://mlsanity.com/paper/2506.00264/) (2025)
- [SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examples](https://mlsanity.com/paper/2305.07984/) (2023)
- [Toward Robust RALMs: Revealing the Impact of Imperfect Retrieval on Retrieval-Augmented Language Models](https://mlsanity.com/paper/2410.15107/) (2024)
- [mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval](https://mlsanity.com/paper/2501.19264/) (2025)
- [DRS: Deep Question Reformulation With Structured Output](https://mlsanity.com/paper/2411.17993/) (2024)
- [Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization](https://mlsanity.com/paper/2208.00579/) (2022)
- [The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models](https://mlsanity.com/paper/2302.00094/) (2023)

## Abstract (arXiv)

Recent pretrained language models "solved" many reading comprehension benchmarks, where questions are written with access to the evidence document. However, datasets containing information-seeking queries where evidence documents are provided after the queries are written independently remain challenging. We analyze why answering information-seeking queries is more challenging and where their prevalent unanswerabilities arise, on Natural Questions and TyDi QA. Our controlled experiments suggest two headrooms -- paragraph selection and answerability prediction, i.e. whether the paired evidence document contains the answer to the query or not. When provided with a gold paragraph and knowing when to abstain from answering, existing models easily outperform a human annotator. However, predicting answerability itself remains challenging. We manually annotate 800 unanswerable examples across six languages on what makes them challenging to answer. With this new data, we conduct per-category answerability prediction, revealing issues in the current dataset collection as well as task formulation. Together, our study points to avenues for future research in information-seeking question answering, both for dataset creation and model development.