mlsanityData to 7 Oct 2026

Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs

arXiv 2311.09469 ↗First posted 16 Nov 2023cs.CLcited by 93

Not on the map. Why?

Cited by 31 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Resolving ambiguities through interaction is a hallmark of natural language, and modeling this behavior is a core challenge in crafting AI assistants. In this work, we study such behavior in LMs by proposing a task-agnostic framework for resolving ambiguity by asking users clarifying questions. Our framework breaks down this objective into three subtasks: (1) determining when clarification is needed, (2) determining what clarifying question to ask, and (3) responding accurately with the new information gathered through clarification. We evaluate systems across three NLP applications: question answering, machine translation and natural language inference. For the first subtask, we present a novel uncertainty estimation approach, intent-sim, that determines the utility of querying for clarification by estimating the entropy over user intents. Our method consistently outperforms existing uncertainty estimation approaches at identifying predictions that will benefit from clarification. When only allowed to ask for clarification on 10% of examples, our system is able to double the performance gains over randomly selecting examples to clarify. Furthermore, we find that intent-sim is robust, demonstrating improvements across a wide range of NLP tasks and LMs. Together, our work lays foundation for studying clarifying interactions with LMs.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Builds on 20 of its 30 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

20 of 30 references.

References · 20

  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  3. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  5. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  6. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  7. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  8. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  9. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  10. 2020Selective Question Answering under Domain Shiftcited by 228
  11. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 155
  12. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 124
8 more
  1. 2018Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Informationcited by 110
  2. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  3. 2017Evaluating Discourse Phenomena in Neural Machine Translationcited by 87
  4. 2019Selective Attention for Context-aware Neural Machine Translationcited by 60
  5. 2021When Does Translation Require Context? A Data-driven, Multilingual Explorationcited by 36
  6. 2019Interactive Classification by Asking Informative Questionscited by 16
  7. 2023Interactive-Chain-Prompting: Ambiguity Resolution for Crosslingual Conditional Generation with Interactioncited by 16
  8. 2023AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreferencecited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 32; 2026: 44.

93 papers.

Sort:

By area:

  1. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 126
  2. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 117
  3. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
  4. 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 70
  5. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 56
  6. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  7. 2025Creating General User Models from Computer Usecited by 44
  8. 2025Structured Uncertainty guided Clarification for LLM Agentscited by 34
  9. 2025What Prompts Don't Say: Understanding and Managing Underspecification in LLM Promptscited by 32
  10. 2025Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineeringcited by 28
  11. 2025Navigating Rifts in Human-LLM Grounding: Study and Benchmarkcited by 27
  12. 2025Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agentscited by 27
38 more
  1. 2024What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Usecited by 24
  2. 2024AmbigDocs: Reasoning across Documents on Different Entities under the Same Namecited by 21
  3. 2025DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMscited by 21
  4. 2025Semantic Volume: Quantifying and Detecting both External and Internal Uncertainty in LLMscited by 19
  5. 2025User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signalcited by 19
  6. 2024STYLE: Improving Domain Transferability of Asking Clarification Questions in Large Language Model Powered Conversational Agentscited by 17
  7. 2026Calibrate-Then-Act: Cost-Aware Exploration in LLM Agentscited by 16
  8. 2024Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citationscited by 15
  9. 2026Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agentscited by 15
  10. 2024ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agentscited by 14
  11. 2024Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Samplingcited by 12
  12. 2024Contextualized Evaluations: Judging Language Model Responses to Underspecified Queriescited by 12
  13. 2025AskToAct: Enhancing LLMs Tool Use via Self-Correcting Clarificationcited by 12
  14. 2025ClarifyMT-Bench: Benchmarking and Improving Multi-Turn Clarification for Conversational Large Language Modelscited by 12
  15. 2025A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directionscited by 9
  16. 2026Agentic Abstention: Do Agents Know When to Stop Instead of Act?cited by 9
  17. 2024RAcQUEt: Unveiling the Dangers of Overlooked Referential Ambiguity in Visual LLMscited by 8
  18. 2025AmbiK: Dataset of Ambiguous Tasks in Kitchen Environmentcited by 8
  19. 2024Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialoguescited by 7
  20. 2025Shoot First, Ask Questions Later? Building Rational Agents that Explore and Act Like Peoplecited by 7
  21. 2025Disambiguation in Conversational Question Answering in the Era of LLMs and Agents: A Surveycited by 6
  22. 2024DRS: Deep Question Reformulation With Structured Outputcited by 5
  23. 2025Referential ambiguity and clarification requests: comparing human and LLM behaviourcited by 5
  24. 2025CaRT: Teaching LLM Agents to Know When They Know Enoughcited by 5
  25. 2025Learning Steerable Clarification Policies with Collaborative Self-playcited by 5
  26. 2026LHAW: Controllable Underspecification for Long-Horizon Taskscited by 5
  27. 2026Safeguarding LLM Agents from Misalignment through Provenance Analysiscited by 4
  28. 2024Comparing Apples to Oranges: LLM-powered Multimodal Intention Prediction in an Object Categorization Taskcited by 3
  29. 2024Situated Instruction Followingcited by 3
  30. 2025Interactive Data Harmonization with LLM Agents: Opportunities and Challengescited by 3
  31. 2025Characterizing Language Use in a Collaborative Situated Gamecited by 3
  32. 2026DiscoverLLM: From Executing Intents to Discovering Themcited by 3
  33. 2026Asking What Matters: Reward-Driven Clarification for Software Engineering Taskscited by 3
  34. 2026Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversationscited by 3
  35. 2024Variability Need Not Imply Error: The Case of Adequate but Semantically Distinct Responsescited by 2
  36. 2025Decide less, communicate more: On the construct validity of end-to-end fact-checking in medicinecited by 2
  37. 2025Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs More Realistic and Less Riskycited by 2
  38. 2026Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requestscited by 2

50 most cited of 93 All 93 (JSON)