mlsanityData to 14 Oct 2026

Disambiguation in Conversational Question Answering in the Era of LLMs and Agents: A Survey

arXiv 2505.12543 ↗First posted 18 May 2025cs.CLcited by 6

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Ambiguity remains a fundamental challenge in Natural Language Processing (NLP) due to the inherent complexity and flexibility of human language. With the advent of Large Language Models (LLMs), addressing ambiguity has become even more critical due to their expanded capabilities and applications. In the context of Conversational Question Answering (CQA), this paper explores the definition, forms, and implications of ambiguity for language driven systems, particularly in the context of LLMs. We define key terms and concepts, categorize various disambiguation approaches enabled by LLMs, and provide a comparative analysis of their advantages and disadvantages. We also explore publicly available datasets for benchmarking ambiguity detection and resolution techniques and highlight their relevance for ongoing research. Finally, we identify open problems and future research directions, especially in agentic settings, proposing areas for further investigation. By offering a comprehensive review of current research on ambiguities and disambiguation with LLMs, we aim to contribute to the development of more robust and reliable LLM-based systems.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 44 of its 84 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

44 of 84 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks7 Nov
  2. 2024Agent-as-a-Judge: Evaluate Agents with Agents14 Oct
  3. 2024Aligning Language Models to Explicitly Handle Ambiguity18 Apr

Other references · 41

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2024A Survey on LLM-as-a-Judgecited by 1,423
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  8. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  9. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  10. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  11. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  12. 2020Open-Domain Question Answering Goes Conversational via Question Rewritingcited by 148
29 more
  1. 2019Asking Clarifying Questions in Open-Domain Information-Seeking Conversationscited by 139
  2. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 126
  3. 2023Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaborationcited by 116
  4. 2018Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Informationcited by 115
  5. 2023Clarify When Necessary: Resolving Ambiguity Through Interaction with LMscited by 98
  6. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 96
  7. 2020Question Rewriting for Conversational Question Answeringcited by 93
  8. 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 76
  9. 2022CLAM: Selective Clarification for Ambiguous Questions with Generative Language Modelscited by 69
  10. 2023Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewritingcited by 69
  11. 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
  12. 2023Large Language Model based Long-tail Query Rewriting in Taobao Searchcited by 62
  13. 2020ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)cited by 58
  14. 2024RaFe: Ranking Feedback Improves Query Rewriting for RAGcited by 49
  15. 2022PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Financecited by 45
  16. 2019Answer-based Adversarial Training for Generating Clarification Questionscited by 43
  17. 2025M-Prometheus: A Suite of Open Multilingual LLM Judgescited by 38
  18. 2020ClarQ: A large-scale and diverse dataset for Clarification Question Generationcited by 31
  19. 2023Asking Clarification Questions to Handle Ambiguity in Open-Domain QAcited by 26
  20. 2020Resolving Intent Ambiguities by Retrieving Discriminative Clarifying Questionscited by 16
  21. 2024Adaptive Query Rewriting: Aligning Rewriters through Marginal Probability of Conversational Answerscited by 15
  22. 2024Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citationscited by 15
  23. 2023Context Aware Query Rewriting for Text Rankers using LLMcited by 14
  24. 2019Identifying Unclear Questions in Community Question Answering Websitescited by 12
  25. 2022Query Refinement Prompts for Closed-Book Long-Form Question Answeringcited by 12
  26. 2024Diversify-verify-adapt: Efficient and Robust Retrieval-Augmented Ambiguous Question Answeringcited by 10
  27. 2025Detecting Ambiguities to Guide Query Rewrite for Robust Conversations in Enterprise AI Assistantscited by 8
  28. 2023LLM Based Multi-Document Summarization Exploiting Main-Event Biased Monotone Submodular Content Extractioncited by 7
  29. 2025Agentic Verification for Ambiguous Query Disambiguationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025A Survey of the State-of-the-Art in Conversational Question Answering Systemscited by 1
  2. 2025SEAL: Self-Evolving Agentic Learning for Conversational Question Answering over Knowledge Graphscited by 1
  3. 2026Who is the richest club in the championship? Detecting and Rewriting Underspecified Questions Improve QA Performancecited by 1
  4. 2026Clarify, Abstain or Answer? Strategising in Conversation with Belief-Augmented Generationcited by 1
  5. 2026CA-BED: Conversation-Aware Bayesian Experimental Designcited by 0
  6. 2026Fewer Clarifications, Better Code: Benchmarking Cross-Session Personalized Ambiguity Adaptation in Coding Assistantscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Ambiguity in LLMs is a concept missing problem
  2. 2025Agent-Based Detection and Resolution of Incompleteness and Ambiguity in Interactions with Large Language Models
  3. 2025Active Task Disambiguation with LLMs
  4. 2026A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models
  5. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Study
  6. 2025An Empirical Study of the Role of Incompleteness and Ambiguity in Interactions with Large Language Models
  7. 2026Small Language Model Helps Resolve Semantic Ambiguity of LLM Prompt
  8. 2024CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models