# CLARA: Classifying and Disambiguating User Commands for Reliable Interactive Robotic Agents

arXiv: https://arxiv.org/abs/2306.10376 · HTML: https://mlsanity.com/paper/2306.10376/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 17 Jun 2023.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [Simple Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2205.06230/) (2022)
- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [We're Afraid Language Models Aren't Modeling Ambiguity](https://mlsanity.com/paper/2304.14399/) (2023)
- [Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis](https://mlsanity.com/paper/2210.04714/) (2022)
- [Quantifying Uncertainties in Natural Language Processing Tasks](https://mlsanity.com/paper/1811.07253/) (2018)
- [InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrieval](https://mlsanity.com/paper/2301.01820/) (2023)
- [Interactive Robotic Grasping with Attribute-Guided Disambiguation](https://mlsanity.com/paper/2203.08037/) (2022)
- [Ask4Help: Learning to Leverage an Expert for Embodied Tasks](https://mlsanity.com/paper/2211.09960/) (2022)
- [DoRO: Disambiguation of referred object for embodied agents](https://mlsanity.com/paper/2207.14205/) (2022)
- [Interactive Disambiguation for Behavior Tree Execution](https://mlsanity.com/paper/2203.02994/) (2022)

## Built on by (most cited first, 30 of 30)

- [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments](https://mlsanity.com/paper/2409.05865/) (2024)
- [Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering](https://mlsanity.com/paper/2502.13069/) (2025)
- [Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Art](https://mlsanity.com/paper/2403.16527/) (2024)
- [Generative Multi-Agent Collaboration in Embodied AI: A Systematic Review](https://mlsanity.com/paper/2502.11518/) (2025)
- [AmbiK: Dataset of Ambiguous Tasks in Kitchen Environment](https://mlsanity.com/paper/2506.04089/) (2025)
- [Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialogues](https://mlsanity.com/paper/2412.01250/) (2024)
- [Imagining a Future of Designing with AI: Dynamic Grounding, Constructive Negotiation, and Sustainable Motivation](https://mlsanity.com/paper/2402.07342/) (2024)
- [REI-Bench: Can Embodied Agents Understand Vague Human Instructions in Task Planning?](https://mlsanity.com/paper/2505.10872/) (2025)
- [Do What? Teaching Vision-Language-Action Models to Reject the Impossible](https://mlsanity.com/paper/2508.16292/) (2025)
- [ADAPT: Actively Discovering and Adapting to Preferences for any Task](https://mlsanity.com/paper/2504.04040/) (2025)
- [A Survey on AgentOps: Categorization, Challenges, and Future Directions](https://mlsanity.com/paper/2508.02121/) (2025)
- [Can only LLMs do Reasoning?: Potential of Small Language Models in Task Planning](https://mlsanity.com/paper/2404.03891/) (2024)
- [Integrating Disambiguation and User Preferences into Large Language Models for Robot Motion Planning](https://mlsanity.com/paper/2404.14547/) (2024)
- [To Ask or Not to Ask? Detecting Absence of Information in Vision and Language Navigation](https://mlsanity.com/paper/2411.05831/) (2024)
- [Characterizing the Robustness of Black-Box LLM Planners Under Perturbed Observations with Adaptive Stress Testing](https://mlsanity.com/paper/2505.05665/) (2025)
- [Extending Epistemic Uncertainty Beyond Parameters Would Assist in Designing Reliable LLMs](https://mlsanity.com/paper/2506.07448/) (2025)
- [Take That for Me: Multimodal Exophora Resolution with Interactive Questioning for Ambiguous Out-of-View Instructions](https://mlsanity.com/paper/2508.16143/) (2025)
- [Toward Ownership Understanding of Objects: Active Question Generation with Large Language Model and Probabilistic Generative Model](https://mlsanity.com/paper/2509.12754/) (2025)
- [Follow-Up Questions Improve Documents Generated by Large Language Models](https://mlsanity.com/paper/2407.12017/) (2024)
- [HELP: Hierarchical Embodied Language Planner for Household Tasks](https://mlsanity.com/paper/2512.21723/) (2025)
- [Hierarchical Compositionality for An Assistive AI Agent](https://mlsanity.com/paper/2608.10330/) (2026)
- [Active Perception for Embodied Disambiguation](https://mlsanity.com/paper/2608.13605/) (2026)
- [RefGuard: Identity-Aware Language-Guided Robot Manipulation via Joint Target-Anchor-Frame Grounding](https://mlsanity.com/paper/2609.06221/) (2026)
- [Robots That Take Initiative: A Framework for Building and Evaluating Proactive Robots](https://mlsanity.com/paper/2609.28910/) (2026)
- [ConflictVLA-Bench: Benchmarking Behavioral Responses of Vision-Language-Action Models to Premise Conflicts](https://mlsanity.com/paper/2609.31792/) (2026)
- [TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns](https://mlsanity.com/paper/2609.38371/) (2026)
- [OGAM: Connecting Systematic Testing to Runtime Assurance through Object-Grounded Attention Monitoring for VLA Policies](https://mlsanity.com/paper/2610.05878/) (2026)
- [MIND: Instruction-Conditioned Incremental Grounding for Robots](https://mlsanity.com/paper/2610.19262/) (2026)

## Abstract (arXiv)

In this paper, we focus on inferring whether the given user command is clear, ambiguous, or infeasible in the context of interactive robotic agents utilizing large language models (LLMs). To tackle this problem, we first present an uncertainty estimation method for LLMs to classify whether the command is certain (i.e., clear) or not (i.e., ambiguous or infeasible). Once the command is classified as uncertain, we further distinguish it between ambiguous or infeasible commands leveraging LLMs with situational aware context in a zero-shot manner. For ambiguous commands, we disambiguate the command by interacting with users via question generation with LLMs. We believe that proper recognition of the given commands could lead to a decrease in malfunction and undesired actions of the robot, enhancing the reliability of interactive robot agents. We present a dataset for robotic situational awareness, consisting pair of high-level commands, scene descriptions, and labels of command type (i.e., clear, ambiguous, or infeasible). We validate the proposed method on the collected dataset, pick-and-place tabletop simulation. Finally, we demonstrate the proposed approach in real-world human-robot interaction experiments, i.e., handover scenarios.