# DEEPAMBIGQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness

arXiv: https://arxiv.org/abs/2511.01323 · HTML: https://mlsanity.com/paper/2511.01323/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 3 Nov 2025.

## Builds on (most cited first, 15 of 15)

- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [The Web as a Knowledge-base for Answering Complex Questions](https://mlsanity.com/paper/1803.06643/) (2018)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [ASQA: Factoid Questions Meet Long-Form Answers](https://mlsanity.com/paper/2204.06092/) (2022)
- [Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases](https://mlsanity.com/paper/2011.07743/) (2020)
- [Variational Reasoning for Question Answering with Knowledge Graph](https://mlsanity.com/paper/1709.04071/) (2017)
- [FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research](https://mlsanity.com/paper/2405.13576/) (2024)
- [KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base](https://mlsanity.com/paper/2007.03875/) (2020)
- [ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubrics](https://mlsanity.com/paper/2509.00496/) (2025)
- [MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents](https://mlsanity.com/paper/2508.11133/) (2025)
- [CondAmbigQA: A Benchmark and Dataset for Conditional Ambiguous Question Answering](https://mlsanity.com/paper/2502.01523/) (2025)
- [WebQuest: A Benchmark for Multimodal QA on Web Page Sequences](https://mlsanity.com/paper/2409.13711/) (2024)

## Built on by (most cited first, 6 of 6)

- [Over-Searching in Search-Augmented Large Language Models](https://mlsanity.com/paper/2601.05503/) (2026)
- [DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agent](https://mlsanity.com/paper/2603.01152/) (2026)
- [When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search](https://mlsanity.com/paper/2606.27669/) (2026)
- [Magic, Madness, Heaven, Sin: LLM Output Diversity is Everything, Everywhere, All at Once](https://mlsanity.com/paper/2604.01504/) (2026)
- [Navigating the Clutter: Waypoint-Based Bi-Level Planning for Multi-Robot Systems](https://mlsanity.com/paper/2604.21138/) (2026)
- [Automating Multi-Hop RAG Evaluation via TRIAD: From Context Extraction to Validated Dataset Generation](https://mlsanity.com/paper/2608.21558/) (2026)

## Abstract (arXiv)

Large language models (LLMs) with integrated search tools show strong promise in open-domain question answering (QA), yet they often struggle to produce complete answer set to complex questions such as Which actor from the film Heat won at least one Academy Award?, which requires (1) distinguishing between multiple films sharing the same title and (2) reasoning across a large set of actors to gather and integrate evidence. Existing QA benchmarks rarely evaluate both challenges jointly. To address this, we introduce DeepAmbigQAGen, an automatic data generation pipeline that constructs QA tasks grounded in text corpora and linked knowledge graph, generating natural and verifiable questions that systematically embed name ambiguity and multi-step reasoning. Based on this, we build DeepAmbigQA, a dataset of 3,600 questions requiring multi-hop reasoning and half of them explicit name ambiguity resolving. Experiments reveal that, even state-of-the-art GPT-5 show incomplete answers, achieving only 0.13 exact match on ambiguous questions and 0.21 on non-ambiguous questions. These findings highlight the need for more robust QA systems aimed at information gathering and answer completeness.