# AmbiK: Dataset of Ambiguous Tasks in Kitchen Environment arXiv: https://arxiv.org/abs/2506.04089 ยท HTML: https://mlsanity.com/paper/2506.04089/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 29 of its 58 references that are in the map. - First posted on arXiv 4 Jun 2025. ## Builds on (most cited first, 29 of 29) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022) - [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022) - [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023) - [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023) - [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021) - [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022) - [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023) - [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021) - [LM-Polygraph: Uncertainty Estimation for Language Models](https://mlsanity.com/paper/2311.07383/) (2023) - [Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness](https://mlsanity.com/paper/2308.16175/) (2023) - [Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs](https://mlsanity.com/paper/2311.09469/) (2023) - [Executing Instructions in Situated Collaborative Interactions](https://mlsanity.com/paper/1910.03655/) (2019) - [API Is Enough: Conformal Prediction for Large Language Models Without Logit-Access](https://mlsanity.com/paper/2403.01216/) (2024) - [Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning](https://mlsanity.com/paper/2312.05230/) (2023) - [ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)](https://mlsanity.com/paper/2009.11352/) (2020) - [SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations](https://mlsanity.com/paper/2104.08667/) (2021) - [A Survey of Uncertainty Estimation in LLMs: Theory Meets Practice](https://mlsanity.com/paper/2410.15326/) (2024) - [Just Ask:An Interactive Learning Framework for Vision and Language Navigation](https://mlsanity.com/paper/1912.00915/) (2019) - [Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions](https://mlsanity.com/paper/2402.06509/) (2024) - [ClarQ: A large-scale and diverse dataset for Clarification Question Generation](https://mlsanity.com/paper/2006.05986/) (2020) - [Think, Act, and Ask: Open-World Interactive Personalized Robot Navigation](https://mlsanity.com/paper/2310.07968/) (2023) - [CLARA: Classifying and Disambiguating User Commands for Reliable Interactive Robotic Agents](https://mlsanity.com/paper/2306.10376/) (2023) - [Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting](https://mlsanity.com/paper/2310.11732/) (2023) - [IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022](https://mlsanity.com/paper/2205.13771/) (2022) - [A recipe for annotating grounded clarifications](https://mlsanity.com/paper/2104.08964/) (2021) - [Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset](https://mlsanity.com/paper/2302.14406/) (2023) - [Risk-Calibrated Human-Robot Interaction via Set-Valued Intent Prediction](https://mlsanity.com/paper/2403.15959/) (2024) - [Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requests](https://mlsanity.com/paper/2401.17039/) (2024) ## Built on by (most cited first, 9 of 9) - [Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents](https://mlsanity.com/paper/2510.17491/) (2025) - [Proactive Rejection and Grounded Execution: A Dual-Stage Intent Analysis Paradigm for Safe and Efficient AIoT Smart Homes](https://mlsanity.com/paper/2603.16207/) (2026) - [HELP: Hierarchical Embodied Language Planner for Household Tasks](https://mlsanity.com/paper/2512.21723/) (2025) - [Learning Transferable Latent User Preferences for Human-Aligned Decision Making](https://mlsanity.com/paper/2605.12682/) (2026) - [Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models](https://mlsanity.com/paper/2606.19297/) (2026) - [Hierarchical Compositionality for An Assistive AI Agent](https://mlsanity.com/paper/2608.10330/) (2026) - [Bayesian Active Learning for Intent Disambiguation in Interactive Robot Planning](https://mlsanity.com/paper/2609.34270/) (2026) - [TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns](https://mlsanity.com/paper/2609.38371/) (2026) - [A Persian Vision-Language-Action Dataset and Benchmark for Ambiguity- and Infeasibility-Aware Instruction-to-Plan Grounding](https://mlsanity.com/paper/2610.15666/) (2026) ## Abstract (arXiv) As a part of an embodied agent, Large Language Models (LLMs) are typically used for behavior planning given natural language instructions from the user. However, dealing with ambiguous instructions in real-world environments remains a challenge for LLMs. Various methods for task ambiguity detection have been proposed. However, it is difficult to compare them because they are tested on different datasets and there is no universal benchmark. For this reason, we propose AmbiK (Ambiguous Tasks in Kitchen Environment), the fully textual dataset of ambiguous instructions addressed to a robot in a kitchen environment. AmbiK was collected with the assistance of LLMs and is human-validated. It comprises 1000 pairs of ambiguous tasks and their unambiguous counterparts, categorized by ambiguity type (Human Preferences, Common Sense Knowledge, Safety), with environment descriptions, clarifying questions and answers, user intents, and task plans, for a total of 2000 tasks. We hope that AmbiK will enable researchers to perform a unified comparison of ambiguity detection methods. AmbiK is available at https://github.com/cog-model/AmbiK-dataset.