# CREPE: Open-Domain Question Answering with False Presuppositions

arXiv: https://arxiv.org/abs/2211.17257 · HTML: https://mlsanity.com/paper/2211.17257/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 53 references that are in the map.
- First posted on arXiv 30 Nov 2022.

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [QuAC : Question Answering in Context](https://mlsanity.com/paper/1808.07036/) (2018)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Hurdles to Progress in Long-form Question Answering](https://mlsanity.com/paper/2103.06332/) (2021)
- [SituatedQA: Incorporating Extra-Linguistic Contexts into QA](https://mlsanity.com/paper/2109.06157/) (2021)
- [Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets](https://mlsanity.com/paper/2008.02637/) (2020)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition](https://mlsanity.com/paper/2004.03066/) (2020)
- [Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval](https://mlsanity.com/paper/2010.11915/) (2020)
- [Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering](https://mlsanity.com/paper/2101.00391/) (2021)

## Built on by (most cited first, 30 of 72)

- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [Can Language Models Solve Graph Problems in Natural Language?](https://mlsanity.com/paper/2305.10037/) (2023)
- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [The Art of Saying No: Contextual Noncompliance in Language Models](https://mlsanity.com/paper/2407.12043/) (2024)
- [Measuring Sycophancy of Language Models in Multi-turn Dialogues](https://mlsanity.com/paper/2505.23840/) (2025)
- [The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models](https://mlsanity.com/paper/2406.05761/) (2024)
- [Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation](https://mlsanity.com/paper/2407.10817/) (2024)
- [(QA)2: Question Answering with Questionable Assumptions](https://mlsanity.com/paper/2212.10003/) (2022)
- [Won't Get Fooled Again: Answering Questions with False Premises](https://mlsanity.com/paper/2307.02394/) (2023)
- [BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information](https://mlsanity.com/paper/2306.07934/) (2023)
- [Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledge](https://mlsanity.com/paper/2311.09731/) (2023)
- [IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions](https://mlsanity.com/paper/2305.14010/) (2023)
- [Enhancing Large Language Models Against Inductive Instructions with Dual-critique Prompting](https://mlsanity.com/paper/2305.13733/) (2023)
- [How to Protect Yourself from 5G Radiation? Investigating LLM Responses to Implicit Misinformation](https://mlsanity.com/paper/2503.09598/) (2025)
- [Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citations](https://mlsanity.com/paper/2410.04241/) (2024)
- [Don't Let It Hallucinate: Premise Verification via Retrieval-Augmented Logical Reasoning](https://mlsanity.com/paper/2504.06438/) (2025)
- [Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositions](https://mlsanity.com/paper/2504.11373/) (2025)
- [Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answering](https://mlsanity.com/paper/2311.09542/) (2023)
- [Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?](https://mlsanity.com/paper/2410.15512/) (2024)
- [Detecting (Un)answerability in Large Language Models with Linear Directions](https://mlsanity.com/paper/2509.22449/) (2025)
- [Evaluating Large Language Models for Health-related Queries with Presuppositions](https://mlsanity.com/paper/2312.08800/) (2023)
- [What makes a good metric? Evaluating automatic metrics for text-to-image consistency](https://mlsanity.com/paper/2412.13989/) (2024)
- [Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skipping](https://mlsanity.com/paper/2503.21817/) (2025)
- [LLMs Struggle to Reject False Presuppositions when Misinformation Stakes are High](https://mlsanity.com/paper/2505.22354/) (2025)
- [Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions](https://mlsanity.com/paper/2506.08952/) (2025)
- [Whispers that Shake Foundations: Analyzing and Mitigating False Premise Hallucinations in Large Language Models](https://mlsanity.com/paper/2402.19103/) (2024)
- [KG-FPQ: Evaluating Factuality Hallucination in LLMs with Knowledge Graph-based False Premise Questions](https://mlsanity.com/paper/2407.05868/) (2024)
- [MultiHoax: A Dataset of Multi-hop False-Premise Questions](https://mlsanity.com/paper/2506.00264/) (2025)
- [Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Models](https://mlsanity.com/paper/2505.23715/) (2025)
- [Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialness](https://mlsanity.com/paper/2406.16342/) (2024)

## Abstract (arXiv)

Information seeking users often pose questions with false presuppositions, especially when asking about unfamiliar topics. Most existing question answering (QA) datasets, in contrast, assume all questions have well defined answers. We introduce CREPE, a QA dataset containing a natural distribution of presupposition failures from online information-seeking forums. We find that 25% of questions contain false presuppositions, and provide annotations for these presuppositions and their corrections. Through extensive baseline experiments, we show that adaptations of existing open-domain QA models can find presuppositions moderately well, but struggle when predicting whether a presupposition is factually correct. This is in large part due to difficulty in retrieving relevant evidence passages from a large text corpus. CREPE provides a benchmark to study question answering in the wild, and our analyses provide avenues for future work in better modeling and further studying the task.