# Don't `Well, Actually' Me Unless You Know What You're Talking About: Weak Presupposition Verification Degrades General QA Performance

arXiv: https://arxiv.org/abs/2608.06539 · HTML: https://mlsanity.com/paper/2608.06539/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 6 Aug 2026.

## Builds on (most cited first, 22 of 22)

- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://mlsanity.com/paper/2404.10774/) (2024)
- [Med42-v2: A Suite of Clinical LLMs](https://mlsanity.com/paper/2408.06142/) (2024)
- [Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge](https://mlsanity.com/paper/2102.03315/) (2021)
- [The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs](https://mlsanity.com/paper/2501.10970/) (2025)
- [CREPE: Open-Domain Question Answering with False Presuppositions](https://mlsanity.com/paper/2211.17257/) (2022)
- [(QA)2: Question Answering with Questionable Assumptions](https://mlsanity.com/paper/2212.10003/) (2022)
- [Won't Get Fooled Again: Answering Questions with False Premises](https://mlsanity.com/paper/2307.02394/) (2023)
- [Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering](https://mlsanity.com/paper/2101.00391/) (2021)
- [Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer](https://mlsanity.com/paper/2502.12964/) (2025)
- [Enhancing Large Language Models Against Inductive Instructions with Dual-critique Prompting](https://mlsanity.com/paper/2305.13733/) (2023)
- [Accounting for Sycophancy in Language Model Uncertainty Estimation](https://mlsanity.com/paper/2410.14746/) (2024)
- [Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositions](https://mlsanity.com/paper/2504.11373/) (2025)
- [Accommodation and Epistemic Vigilance: A Pragmatic Account of Why LLMs Fail to Challenge Harmful Beliefs](https://mlsanity.com/paper/2601.04435/) (2026)
- [Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions](https://mlsanity.com/paper/2506.08952/) (2025)
- [KG-FPQ: Evaluating Factuality Hallucination in LLMs with Knowledge Graph-based False Premise Questions](https://mlsanity.com/paper/2407.05868/) (2024)
- [MultiHoax: A Dataset of Multi-hop False-Premise Questions](https://mlsanity.com/paper/2506.00264/) (2025)
- [Syn-QA2: Evaluating False Assumptions in Long-tail Questions with Synthetic QA Datasets](https://mlsanity.com/paper/2403.12145/) (2024)
- [Two Axes of LLM Abstention: Answer Correctness and Question Answerability](https://mlsanity.com/paper/2607.08456/) (2026)
- [PreWoMe: Exploiting Presuppositions as Working Memory for Long Form Question Answering](https://mlsanity.com/paper/2310.16147/) (2023)
- [PhantomBench: Benchmarking the Non-existential Threat of Language Models](https://mlsanity.com/paper/2606.11105/) (2026)
- [Legal LLM Hallucination Should Be Evaluated as Failure of Legal Warrant](https://mlsanity.com/paper/2609.17546/) (2026)

## Built on by (most cited first, 1 of 1)

- [The Civilization Framework: Sovereign-Anchored Communication Between Personal Multi-Agent Systems](https://mlsanity.com/paper/2609.03425/) (2026)

## Abstract (arXiv)

False-presupposition QA (FPQA) tests LLMs on their ability to identify false presuppositions in questions and abstain or correct them rather than reinforcing false assumptions. The common approach reduces the task to prompting LLMs to extract presuppositions and fact checking each presupposition. While the performance on dedicated benchmarks keeps improving, evaluation largely focuses on questions with false presuppositions (FPQs) while ignoring the performance on ``normal'' questions (TPQs). Since many benchmarks over-represent FPQs compared to their natural occurrence, the result is that performance on these benchmarks doesn't reflect real-world QA performance. Through extensive experiments across various model families, sizes, and benchmarks, we show that methods that perform better on FPQs tend to perform worse on TPQs. Our analysis reveals this is the result of weak fact checking modules that reject also true presuppositions. We hope our findings will help guide future work toward FPQA methods that generalize well to realistic settings.