mlsanityData to 14 Oct 2026

Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions

arXiv 2605.25284 ↗First posted 24 May 2026cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

User queries are often underspecified and may admit multiple valid interpretations. Rather than silently making assumptions about the user's intent, a helpful assistant should surface such ambiguity by asking a clarifying question. Doing so requires two abilities: recognizing that a query is ambiguous, and acting on that recognition by seeking clarification instead of answering directly. To study these abilities, we evaluate models on ambiguous, unambiguous, and disambiguated questions in three settings: standard question answering, explicit ambiguity judgment, and behavioral analysis, where a judge model classifies responses as direct answers, refusals, or clarifying questions. We find a clear gap between recognition and behavior: models often identify ambiguity when explicitly asked to judge it, yet in the QA setting they overwhelmingly default to direct answers. Retrieved context further widens this gap by improving answerability while making models even less likely to ask clarifying questions.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 24 May 2026.

Builds on

20 of 27 references.

References · 20

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  4. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  5. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  6. 2020Selective Question Answering under Domain Shiftcited by 238
  7. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  8. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 158
  9. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  10. 2019Asking Clarifying Questions in Open-Domain Information-Seeking Conversationscited by 139
  11. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 126
  12. 2023Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaborationcited by 116
8 more
  1. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 105
  2. 2022CLAM: Selective Clarification for Ambiguous Questions with Generative Language Modelscited by 69
  3. 2023Selectively Answering Ambiguous Questionscited by 66
  4. 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
  5. 2020ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)cited by 58
  6. 2020ClarQ: A large-scale and diverse dataset for Clarification Question Generationcited by 31
  7. 2023Asking Clarification Questions to Handle Ambiguity in Open-Domain QAcited by 26
  8. 2025MIRAGE: A Metric-Intensive Benchmark for Retrieval-Augmented Generation Evaluationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.