# MultiHoax: A Dataset of Multi-hop False-Premise Questions

arXiv: https://arxiv.org/abs/2506.00264 · HTML: https://mlsanity.com/paper/2506.00264/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 53 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 30 of 32)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries](https://mlsanity.com/paper/2401.15391/) (2024)
- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [Do Large Language Models Latently Perform Multi-Hop Reasoning?](https://mlsanity.com/paper/2402.16837/) (2024)
- [Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence](https://mlsanity.com/paper/2210.13701/) (2022)
- [CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations](https://mlsanity.com/paper/2310.11501/) (2023)
- [ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic](https://mlsanity.com/paper/2402.12840/) (2024)
- [CREPE: Open-Domain Question Answering with False Presuppositions](https://mlsanity.com/paper/2211.17257/) (2022)
- [CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean](https://mlsanity.com/paper/2403.06412/) (2024)
- [Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking](https://mlsanity.com/paper/2402.09369/) (2024)
- [Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU](https://mlsanity.com/paper/2310.04928/) (2023)
- [(QA)2: Question Answering with Questionable Assumptions](https://mlsanity.com/paper/2212.10003/) (2022)
- [Won't Get Fooled Again: Answering Questions with False Premises](https://mlsanity.com/paper/2307.02394/) (2023)
- [Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval](https://mlsanity.com/paper/2010.11915/) (2020)
- [BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information](https://mlsanity.com/paper/2306.07934/) (2023)
- [Faking Fake News for Real Fake News Detection: Propaganda-loaded Training Data Generation](https://mlsanity.com/paper/2203.05386/) (2022)
- [How susceptible are LLMs to Logical Fallacies?](https://mlsanity.com/paper/2308.09853/) (2023)
- [LLM-Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environments](https://mlsanity.com/paper/2408.15903/) (2024)
- [Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citations](https://mlsanity.com/paper/2410.04241/) (2024)
- [Logical Consistency of Large Language Models in Fact-checking](https://mlsanity.com/paper/2412.16100/) (2024)
- [Failure Modes of LLMs for Causal Reasoning on Narratives](https://mlsanity.com/paper/2410.23884/) (2024)
- [Characterizing Multi-Domain False News and Underlying User Effects on Chinese Weibo](https://mlsanity.com/paper/2205.03068/) (2022)
- [Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Norms](https://mlsanity.com/paper/2310.10418/) (2023)
- [Toward Robust RALMs: Revealing the Impact of Imperfect Retrieval on Retrieval-Augmented Language Models](https://mlsanity.com/paper/2410.15107/) (2024)
- [Can I introduce my boyfriend to my grandmother? Evaluating Large Language Models Capabilities on Iranian Social Norm Classification](https://mlsanity.com/paper/2406.09123/) (2024)

## Built on by (most cited first, 9 of 9)

- [Conflict-Aware Fusion: Mitigating Logic Inertia in Large Language Models via Structured Cognitive Priors](https://mlsanity.com/paper/2512.06393/) (2025)
- [BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidence](https://mlsanity.com/paper/2603.07931/) (2026)
- [LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection](https://mlsanity.com/paper/2604.04815/) (2026)
- [Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs](https://mlsanity.com/paper/2605.13737/) (2026)
- [HopRefusalBench: Diagnosing Refusal Failures in Search-Augmented Agents for Multi-Hop Reasoning](https://mlsanity.com/paper/2608.01358/) (2026)
- [Don't `Well, Actually' Me Unless You Know What You're Talking About: Weak Presupposition Verification Degrades General QA Performance](https://mlsanity.com/paper/2608.06539/) (2026)
- [CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild](https://mlsanity.com/paper/2605.30241/) (2026)
- [XYEval: Agents say yes to bad advice](https://mlsanity.com/paper/2609.23939/) (2026)
- [Pushback in a Haystack: Language Models Respond to Factual Errors Ambiguously and Ineffectively](https://mlsanity.com/paper/2610.17703/) (2026)

## Abstract (arXiv)

As Large Language Models are increasingly deployed in high-stakes domains, their ability to detect false assumptions and reason critically is crucial for ensuring reliable outputs. False-premise questions (FPQs) serve as an important evaluation method by exposing cases where flawed assumptions lead to incorrect responses. While existing benchmarks focus on single-hop FPQs, real-world reasoning often requires multi-hop inference, where models must verify consistency across multiple reasoning steps rather than relying on surface-level cues. To address this gap, we introduce MultiHoax, a benchmark for evaluating LLMs' ability to handle false premises in complex, multi-step reasoning tasks. Our dataset spans seven countries and ten diverse knowledge categories, using Wikipedia as the primary knowledge source to enable factual reasoning across regions. Experiments reveal that state-of-the-art LLMs struggle to detect false premises across different countries, knowledge categories, and multi-hop reasoning types, highlighting the need for improved false premise detection and more robust multi-hop reasoning capabilities in LLMs.