mlsanityData to 14 Oct 2026

AmbiK: Dataset of Ambiguous Tasks in Kitchen Environment

arXiv 2506.04089 ↗First posted 4 Jun 2025cs.LG cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

As a part of an embodied agent, Large Language Models (LLMs) are typically used for behavior planning given natural language instructions from the user. However, dealing with ambiguous instructions in real-world environments remains a challenge for LLMs. Various methods for task ambiguity detection have been proposed. However, it is difficult to compare them because they are tested on different datasets and there is no universal benchmark. For this reason, we propose AmbiK (Ambiguous Tasks in Kitchen Environment), the fully textual dataset of ambiguous instructions addressed to a robot in a kitchen environment. AmbiK was collected with the assistance of LLMs and is human-validated. It comprises 1000 pairs of ambiguous tasks and their unambiguous counterparts, categorized by ambiguity type (Human Preferences, Common Sense Knowledge, Safety), with environment descriptions, clarifying questions and answers, user intents, and task plans, for a total of 2000 tasks. We hope that AmbiK will enable researchers to perform a unified comparison of ambiguity detection methods. AmbiK is available at https://github.com/cog-model/AmbiK-dataset.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 29 of its 58 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

29 of 58 references.

References · 29

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  4. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  5. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  6. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  7. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  8. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
  9. 2021TEACh: Task-driven Embodied Agents that Chatcited by 197
  10. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 131
  11. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 122
  12. 2023Clarify When Necessary: Resolving Ambiguity Through Interaction with LMscited by 98
17 more
  1. 2019Executing Instructions in Situated Collaborative Interactionscited by 69
  2. 2024API Is Enough: Conformal Prediction for Large Language Models Without Logit-Accesscited by 69
  3. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 60
  4. 2020ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)cited by 58
  5. 2021SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversationscited by 56
  6. 2024A Survey of Uncertainty Estimation in LLMs: Theory Meets Practicecited by 55
  7. 2019Just Ask:An Interactive Learning Framework for Vision and Language Navigationcited by 53
  8. 2024Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questionscited by 32
  9. 2020ClarQ: A large-scale and diverse dataset for Clarification Question Generationcited by 31
  10. 2023Think, Act, and Ask: Open-World Interactive Personalized Robot Navigationcited by 31
  11. 2023CLARA: Classifying and Disambiguating User Commands for Reliable Interactive Robotic Agentscited by 30
  12. 2023Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Settingcited by 20
  13. 2022IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022cited by 14
  14. 2021A recipe for annotating grounded clarificationscited by 13
  15. 2023Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Datasetcited by 11
  16. 2024Risk-Calibrated Human-Robot Interaction via Set-Valued Intent Predictioncited by 9
  17. 2024Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requestscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  2. 2026Proactive Rejection and Grounded Execution: A Dual-Stage Intent Analysis Paradigm for Safe and Efficient AIoT Smart Homescited by 1
  3. 2025HELP: Hierarchical Embodied Language Planner for Household Taskscited by 0
  4. 2026Learning Transferable Latent User Preferences for Human-Aligned Decision Makingcited by 0
  5. 2026Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Modelscited by 0
  6. 2026Hierarchical Compositionality for An Assistive AI Agentcited by 0
  7. 2026Bayesian Active Learning for Intent Disambiguation in Interactive Robot Planningcited by 0
  8. 2026TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patternscited by 0
  9. 2026A Persian Vision-Language-Action Dataset and Benchmark for Ambiguity- and Infeasibility-Aware Instruction-to-Plan Groundingcited by 0