mlsanityData to 14 Oct 2026

CREPE: Open-Domain Question Answering with False Presuppositions

arXiv 2211.17257 ↗First posted 30 Nov 2022cs.CL cs.AIcited by 72

Not on the map. Why?

Abstract (arXiv)

Information seeking users often pose questions with false presuppositions, especially when asking about unfamiliar topics. Most existing question answering (QA) datasets, in contrast, assume all questions have well defined answers. We introduce CREPE, a QA dataset containing a natural distribution of presupposition failures from online information-seeking forums. We find that 25% of questions contain false presuppositions, and provide annotations for these presuppositions and their corrections. Through extensive baseline experiments, we show that adaptations of existing open-domain QA models can find presuppositions moderately well, but struggle when predicting whether a presupposition is factually correct. This is in large part due to difficulty in retrieving relevant evidence passages from a large text corpus. CREPE provides a benchmark to study question answering in the wild, and our analyses provide avenues for future work in better modeling and further studying the task.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Builds on 31 of its 53 references that are in the map. First posted on arXiv 30 Nov 2022.

Builds on

31 of 53 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  6. 2017Billion-scale similarity search with GPUscited by 2,497
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  11. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  12. 2017Mixed Precision Trainingcited by 1,156
19 more
  1. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  2. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  3. 2019ELI5: Long Form Question Answeringcited by 575
  4. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  5. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  6. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  7. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  8. 2018QuAC : Question Answering in Contextcited by 381
  9. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 380
  10. 2020Autoregressive Entity Retrievalcited by 334
  11. 2020Evaluation of Text Generation: A Surveycited by 277
  12. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  13. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  14. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 138
  15. 2018A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applicationscited by 136
  16. 2020Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESuppositioncited by 77
  17. 2020Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrievalcited by 44
  18. 2021Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answeringcited by 41
  19. 2022How Do We Answer Complex Questions: Discourse Structure of Long-form Answerscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 12; 2024: 19; 2025: 19; 2026: 21.

72 papers.

Sort:

By area:

  1. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  2. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 252
  3. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  4. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 105
  5. 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 104
  6. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  7. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 74
  8. 2022(QA)2: Question Answering with Questionable Assumptionscited by 51
  9. 2023Won't Get Fooled Again: Answering Questions with False Premisescited by 48
  10. 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Informationcited by 44
  11. 2023Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledgecited by 32
  12. 2023IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositionscited by 26
38 more
  1. 2023Enhancing Large Language Models Against Inductive Instructions with Dual-critique Promptingcited by 21
  2. 2025How to Protect Yourself from 5G Radiation? Investigating LLM Responses to Implicit Misinformationcited by 18
  3. 2024Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citationscited by 15
  4. 2025Don't Let It Hallucinate: Premise Verification via Retrieval-Augmented Logical Reasoningcited by 15
  5. 2025Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositionscited by 15
  6. 2023Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answeringcited by 14
  7. 2024Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?cited by 13
  8. 2025Detecting (Un)answerability in Large Language Models with Linear Directionscited by 13
  9. 2023Evaluating Large Language Models for Health-related Queries with Presuppositionscited by 12
  10. 2024What makes a good metric? Evaluating automatic metrics for text-to-image consistencycited by 12
  11. 2025Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skippingcited by 12
  12. 2025LLMs Struggle to Reject False Presuppositions when Misinformation Stakes are Highcited by 12
  13. 2025Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questionscited by 10
  14. 2024Whispers that Shake Foundations: Analyzing and Mitigating False Premise Hallucinations in Large Language Modelscited by 9
  15. 2024KG-FPQ: Evaluating Factuality Hallucination in LLMs with Knowledge Graph-based False Premise Questionscited by 9
  16. 2025MultiHoax: A Dataset of Multi-hop False-Premise Questionscited by 9
  17. 2025Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Modelscited by 8
  18. 2024Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialnesscited by 7
  19. 2024I Could've Asked That: Reformulating Unanswerable Questionscited by 6
  20. 2024Do great minds think alike? Investigating Human-AI Complementarity in Question Answering with CAIMIRAcited by 6
  21. 2024DRS: Deep Question Reformulation With Structured Outputcited by 6
  22. 2023HOP, UNION, GENERATE: Explainable Multi-hop Reasoning without Rationale Supervisioncited by 5
  23. 2024Syn-QA2: Evaluating False Assumptions in Long-tail Questions with Synthetic QA Datasetscited by 5
  24. 2025GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibrationcited by 5
  25. 2026Two Axes of LLM Abstention: Answer Correctness and Question Answerabilitycited by 5
  26. 2023PreWoMe: Exploiting Presuppositions as Working Memory for Long Form Question Answeringcited by 4
  27. 2024Unanswerability Evaluation for Retrieval Augmented Generationcited by 4
  28. 2024Towards Unbiased Evaluation of Detecting Unanswerable Questions in EHRSQLcited by 3
  29. 2025UAQFact: Evaluating Factual Knowledge Utilization of LLMs on Unanswerable Questionscited by 3
  30. 2026SPINE: Measuring LLM Sycophancy under Sustained Multi-Turn Pressurecited by 3
  31. 2024ELOQ: Resources for Enhancing LLM Detection of Out-of-Scope Questionscited by 2
  32. 2025Visual Perturbation and Adaptive Hard Negative Contrastive Learning for Compositional Reasoning in Vision-Language Modelscited by 2
  33. 2025Large Language Models Require Curated Context for Reliable Political Fact-Checking -- Even with Reasoning and Web Searchcited by 2
  34. 2026Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMscited by 2
  35. 2024How the Advent of Ubiquitous Large Language Models both Stymie and Turbocharge Dynamic Adversarial Question Generationcited by 1
  36. 2024Interpreting Answers to Yes-No Questions in Dialogues from Multiple Domainscited by 1
  37. 2024Towards Robust Evaluation: A Comprehensive Taxonomy of Datasets and Metrics for Open Domain Question Answering in the Era of Large Language Modelscited by 1
  38. 2025How do Transformer Embeddings Represent Compositions? A Functional Analysiscited by 1

50 most cited of 72 All 72 (JSON)