mlsanityData to 14 Oct 2026

CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models

arXiv 2405.12063 ↗First posted 20 May 2024cs.CLcited by 54

Not on the map. Why?

Cited by 20 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) are increasingly used to meet user information needs, but their effectiveness in dealing with user queries that contain various types of ambiguity remains unknown, ultimately risking user trust and satisfaction. To this end, we introduce CLAMBER, a benchmark for evaluating LLMs using a well-organized taxonomy. Building upon the taxonomy, we construct ~12K high-quality data to assess the strengths, weaknesses, and potential risks of various off-the-shelf LLMs. Our findings indicate the limited practical utility of current LLMs in identifying and clarifying ambiguous user queries, even enhanced by chain-of-thought (CoT) and few-shot prompting. These techniques may result in overconfidence in LLMs and yield only marginal enhancements in identifying ambiguity. Furthermore, current LLMs fall short in generating high-quality clarifying questions due to a lack of conflict resolution and inaccurate utilization of inherent knowledge. In this paper, CLAMBER presents a guidance and promotes further research on proactive and trustworthy LLMs. Our dataset is available at https://github.com/zt991211/CLAMBER

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Builds on 16 of its 48 references that are in the map. First posted on arXiv 20 May 2024.

Builds on

16 of 48 references.

References · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  3. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  4. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  5. 2023Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaborationcited by 116
  6. 2022CLAM: Selective Clarification for Ambiguous Questions with Generative Language Modelscited by 69
  7. 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
  8. 2023Do Large Language Models Know about Facts?cited by 51
  9. 2023ALCUNA: Large Language Models Meet New Knowledgecited by 35
  10. 2021Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLPcited by 31
  11. 2023A Survey on Asking Clarification Questions Datasets in Conversational Systemscited by 28
  12. 2023Asking Clarification Questions to Handle Ambiguity in Open-Domain QAcited by 26
4 more
  1. 2019Identifying Unclear Questions in Community Question Answering Websitescited by 12
  2. 2023AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreferencecited by 12
  3. 2024Concept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factorscited by 9
  4. 2022ORCAS-I: Queries Annotated with Intent using Weak Supervisioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 17; 2026: 35.

54 papers.

Sort:

By area:

  1. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  2. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 63
  3. 2025Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineeringcited by 38
  4. 2025What Prompts Don't Say: Understanding and Managing Underspecification in LLM Promptscited by 32
  5. 2025Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agentscited by 32
  6. 2026HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?cited by 27
  7. 2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Conditioncited by 23
  8. 2025Semantic Volume: Quantifying and Detecting both External and Internal Uncertainty in LLMscited by 20
  9. 2026AgentAbstain: Do LLM Agents Know When Not to Act?cited by 17
  10. 2026Agentic Abstention: Do Agents Know When to Stop Instead of Act?cited by 15
  11. 2025Geometric Uncertainty for Detecting and Correcting Hallucinations in LLMscited by 13
  12. 2024Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?cited by 8
38 more
  1. 2025CaRT: Teaching LLM Agents to Know When They Know Enoughcited by 7
  2. 2025Towards Evaluating Proactive Risk Awareness of Multimodal Language Modelscited by 6
  3. 2026Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environmentscited by 6
  4. 2026Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agentscited by 5
  5. 2025Zero-Shot and Efficient Clarification Need Prediction in Conversational Searchcited by 4
  6. 2025Can Large Language Models Really Recognize Your Name?cited by 4
  7. 2025RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generationcited by 4
  8. 2026AI Alignment Breaks at the Edgecited by 4
  9. 2026Asking What Matters: Reward-Driven Clarification for Software Engineering Taskscited by 4
  10. 2026The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agentscited by 4
  11. 2025CLEAR-KGQA: Clarification-Enhanced Ambiguity Resolution for Knowledge Graph Question Answeringcited by 3
  12. 2025ConDABench: Interactive Evaluation of Language Models for Data Analysiscited by 3
  13. 2026Ambiguity Collapse by LLMs: A Taxonomy of Epistemic Riskscited by 3
  14. 2025Decide less, communicate more: On the construct validity of end-to-end fact-checking in medicinecited by 2
  15. 2026SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Sciencecited by 2
  16. 2026Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM Agentscited by 2
  17. 2025A Data Synthesis Method Driven by Large Language Models for Proactive Mining of Implicit User Intentions in Tourismcited by 1
  18. 2025From Parameters to Prompts: Understanding and Mitigating the Factuality Gap between Fine-Tuned LLMscited by 1
  19. 2026User Prompting Strategies and Prompt Enhancement Methods for Open-Set Object Detection in XR Environmentscited by 1
  20. 2026Act or Clarify? Modeling Sensitivity to Uncertainty and Cost in Communicationcited by 1
  21. 2026Generative AI for Managerial Decision-Making under Ambiguity and Sycophancycited by 1
  22. 2026InsufficiencyBench: Evaluating LLM legal advice on underspecified user queriescited by 1
  23. 2024Follow-Up Questions Improve Documents Generated by Large Language Modelscited by 0
  24. 2026Agent System Operations: Categorization, Challenges, and Future Directionscited by 0
  25. 2026TIPS: Topological Ill-Posedness Probing and Steering in Large Language Modelscited by 0
  26. 2026Automatically Evolving Prompt Guidelines for Task-Specific Optimizationcited by 0
  27. 2026The Severance Problem: LLMs are Unaware of the Person Beyond the Promptcited by 0
  28. 2026Understanding From Human Perspective: A Multi-agent System for Interactive Egocentric Medical Image Segmentationcited by 0
  29. 2026CITBench: A Comprehensive Benchmark for Interactive Tabular Data Processing with LLMscited by 0
  30. 2026PredAct-Bench: Benchmarking Tool-Augmented Dialogue under Controlled Tool Noisecited by 0
  31. 2026Active Inference as Context Acquisition for AI Agentscited by 0
  32. 2026A knowledge-guided agentic framework for mitigating patient-context ambiguity in health queriescited by 0
  33. 2026FinRiskAtlas: Decision-Aligned Evaluation of Large Language Models for Financial Risk Reviewcited by 0
  34. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0
  35. 2026CruxBench: A Benchmark of Information Discoverycited by 0
  36. 2026Foundations of Proactive Agents: Principles, Technical Layers, and Proactivity-Gymcited by 0
  37. 2026DelegationBench: Measuring When AI Agents Should Ask Before Actingcited by 0
  38. 2026CLEAT: Learning Value-Guided Clarification for Interactive Agentscited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AskToAct: Enhancing LLMs Tool Use via Self-Correcting Clarification
  2. 2025ClarifyMT-Bench: Benchmarking and Improving Multi-Turn Clarification for Conversational Large Language Models
  3. 2024To Know or Not To Know? Analyzing Self-Consistency of Large Language Models under Ambiguity
  4. 2025Beyond Single Models: Enhancing LLM Detection of Ambiguity in Requests through Debate
  5. 2025Clarifying Ambiguities: on the Role of Ambiguity Types in Prompting Methods for Clarification Generation
  6. 2024Aligning Language Models to Explicitly Handle Ambiguity
  7. 2025Ambiguity in LLMs is a concept missing problem
  8. 2026A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models