# CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models

arXiv: https://arxiv.org/abs/2405.12063 · HTML: https://mlsanity.com/paper/2405.12063/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 48 references that are in the map.
- First posted on arXiv 20 May 2024.

## Builds on (most cited first, 16 of 16)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves](https://mlsanity.com/paper/2311.04205/) (2023)
- [Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration](https://mlsanity.com/paper/2305.13626/) (2023)
- [CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models](https://mlsanity.com/paper/2212.07769/) (2022)
- [Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions](https://mlsanity.com/paper/2109.05794/) (2021)
- [Do Large Language Models Know about Facts?](https://mlsanity.com/paper/2310.05177/) (2023)
- [ALCUNA: Large Language Models Meet New Knowledge](https://mlsanity.com/paper/2310.14820/) (2023)
- [Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLP](https://mlsanity.com/paper/2106.06830/) (2021)
- [A Survey on Asking Clarification Questions Datasets in Conversational Systems](https://mlsanity.com/paper/2305.15933/) (2023)
- [Asking Clarification Questions to Handle Ambiguity in Open-Domain QA](https://mlsanity.com/paper/2305.13808/) (2023)
- [Identifying Unclear Questions in Community Question Answering Websites](https://mlsanity.com/paper/1901.06168/) (2019)
- [AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference](https://mlsanity.com/paper/2302.00762/) (2023)
- [Concept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factors](https://mlsanity.com/paper/2404.03304/) (2024)
- [ORCAS-I: Queries Annotated with Intent using Weak Supervision](https://mlsanity.com/paper/2205.00926/) (2022)

## Built on by (most cited first, 30 of 54)

- [AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions](https://mlsanity.com/paper/2506.09038/) (2025)
- [QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?](https://mlsanity.com/paper/2503.22674/) (2025)
- [Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering](https://mlsanity.com/paper/2502.13069/) (2025)
- [What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts](https://mlsanity.com/paper/2505.13360/) (2025)
- [Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agents](https://mlsanity.com/paper/2505.22655/) (2025)
- [HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?](https://mlsanity.com/paper/2604.09408/) (2026)
- [AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition](https://mlsanity.com/paper/2602.11348/) (2026)
- [Semantic Volume: Quantifying and Detecting both External and Internal Uncertainty in LLMs](https://mlsanity.com/paper/2502.21239/) (2025)
- [AgentAbstain: Do LLM Agents Know When Not to Act?](https://mlsanity.com/paper/2607.10059/) (2026)
- [Agentic Abstention: Do Agents Know When to Stop Instead of Act?](https://mlsanity.com/paper/2606.28733/) (2026)
- [Geometric Uncertainty for Detecting and Correcting Hallucinations in LLMs](https://mlsanity.com/paper/2509.13813/) (2025)
- [Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?](https://mlsanity.com/paper/2406.12307/) (2024)
- [CaRT: Teaching LLM Agents to Know When They Know Enough](https://mlsanity.com/paper/2510.08517/) (2025)
- [Towards Evaluating Proactive Risk Awareness of Multimodal Language Models](https://mlsanity.com/paper/2505.17455/) (2025)
- [Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environments](https://mlsanity.com/paper/2605.27209/) (2026)
- [Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents](https://mlsanity.com/paper/2605.09698/) (2026)
- [Zero-Shot and Efficient Clarification Need Prediction in Conversational Search](https://mlsanity.com/paper/2503.00179/) (2025)
- [Can Large Language Models Really Recognize Your Name?](https://mlsanity.com/paper/2505.14549/) (2025)
- [RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation](https://mlsanity.com/paper/2508.06056/) (2025)
- [AI Alignment Breaks at the Edge](https://mlsanity.com/paper/2602.20042/) (2026)
- [Asking What Matters: Reward-Driven Clarification for Software Engineering Tasks](https://mlsanity.com/paper/2604.14624/) (2026)
- [The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agents](https://mlsanity.com/paper/2605.20544/) (2026)
- [CLEAR-KGQA: Clarification-Enhanced Ambiguity Resolution for Knowledge Graph Question Answering](https://mlsanity.com/paper/2504.09665/) (2025)
- [ConDABench: Interactive Evaluation of Language Models for Data Analysis](https://mlsanity.com/paper/2510.13835/) (2025)
- [Ambiguity Collapse by LLMs: A Taxonomy of Epistemic Risks](https://mlsanity.com/paper/2603.05801/) (2026)
- [Decide less, communicate more: On the construct validity of end-to-end fact-checking in medicine](https://mlsanity.com/paper/2506.20876/) (2025)
- [SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science](https://mlsanity.com/paper/2605.18630/) (2026)
- [Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM Agents](https://mlsanity.com/paper/2608.19564/) (2026)
- [A Data Synthesis Method Driven by Large Language Models for Proactive Mining of Implicit User Intentions in Tourism](https://mlsanity.com/paper/2505.11533/) (2025)
- [From Parameters to Prompts: Understanding and Mitigating the Factuality Gap between Fine-Tuned LLMs](https://mlsanity.com/paper/2505.23410/) (2025)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used to meet user information needs, but their effectiveness in dealing with user queries that contain various types of ambiguity remains unknown, ultimately risking user trust and satisfaction. To this end, we introduce CLAMBER, a benchmark for evaluating LLMs using a well-organized taxonomy. Building upon the taxonomy, we construct ~12K high-quality data to assess the strengths, weaknesses, and potential risks of various off-the-shelf LLMs. Our findings indicate the limited practical utility of current LLMs in identifying and clarifying ambiguous user queries, even enhanced by chain-of-thought (CoT) and few-shot prompting. These techniques may result in overconfidence in LLMs and yield only marginal enhancements in identifying ambiguity. Furthermore, current LLMs fall short in generating high-quality clarifying questions due to a lack of conflict resolution and inaccurate utilization of inherent knowledge. In this paper, CLAMBER presents a guidance and promotes further research on proactive and trustworthy LLMs. Our dataset is available at https://github.com/zt991211/CLAMBER