# Improving Grounded Natural Language Understanding through Human-Robot Dialog

arXiv: https://arxiv.org/abs/1903.00122 · HTML: https://mlsanity.com/paper/1903.00122/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2019.

## Built on by (most cited first, 25 of 25)

- [Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation](https://mlsanity.com/paper/2303.03480/) (2023)
- ["No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy](https://mlsanity.com/paper/2301.02555/) (2023)
- [Grounding Open-Domain Instructions to Automate Web Support Tasks](https://mlsanity.com/paper/2103.16057/) (2021)
- [Morae: Proactively Pausing UI Agents for User Choices](https://mlsanity.com/paper/2508.21456/) (2025)
- [Natural Language Processing for Smart Healthcare](https://mlsanity.com/paper/2110.15803/) (2021)
- ["Don't forget to put the milk back!" Dataset for Enabling Embodied Agents to Detect Anomalous Situations](https://mlsanity.com/paper/2404.08827/) (2024)
- [Talk-to-Resolve: Combining scene understanding and spatial dialogue to resolve granular task ambiguity for a collocated robot](https://mlsanity.com/paper/2111.11099/) (2021)
- [DoRO: Disambiguation of referred object for embodied agents](https://mlsanity.com/paper/2207.14205/) (2022)
- [Learning to Summarize and Answer Questions about a Virtual Robot's Past Actions](https://mlsanity.com/paper/2306.09922/) (2023)
- [Natural Language Specification of Reinforcement Learning Policies through Differentiable Decision Trees](https://mlsanity.com/paper/2101.07140/) (2021)
- [Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attention](https://mlsanity.com/paper/2203.06822/) (2022)
- [LLMs for Robotic Object Disambiguation](https://mlsanity.com/paper/2401.03388/) (2024)
- [Few-Shot Visual Grounding for Natural Human-Robot Interaction](https://mlsanity.com/paper/2103.09720/) (2021)
- [Deep Learning for Embodied Vision Navigation: A Survey](https://mlsanity.com/paper/2108.04097/) (2021)
- [Improved Inference of Human Intent by Combining Plan Recognition and Language Feedback](https://mlsanity.com/paper/2310.02462/) (2023)
- [SInViG: A Self-Evolving Interactive Visual Agent for Human-Robot Interaction](https://mlsanity.com/paper/2402.11792/) (2024)
- [A Computational Interface to Translate Strategic Intent from Unstructured Language in a Low-Data Setting](https://mlsanity.com/paper/2208.08374/) (2022)
- [Trust in Language Grounding: a new AI challenge for human-robot teams](https://mlsanity.com/paper/2209.02066/) (2022)
- [GenComUI: Exploring Generative Visual Aids as Medium to Support Task-Oriented Human-Robot Communication](https://mlsanity.com/paper/2502.10678/) (2025)
- [Mixed-Initiative Dialog for Human-Robot Collaborative Manipulation](https://mlsanity.com/paper/2508.05535/) (2025)
- [Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning](https://mlsanity.com/paper/2107.09285/) (2021)
- [HomeEmergency -- Using Audio to Find and Respond to Emergencies in the Home](https://mlsanity.com/paper/2504.01089/) (2025)
- [iLearnRobot: An Interactive Learning-Based Multi-Modal Robot with Continuous Improvement](https://mlsanity.com/paper/2507.22896/) (2025)
- [Hierarchical Compositionality for An Assistive AI Agent](https://mlsanity.com/paper/2608.10330/) (2026)
- [STAGE: Diagnosing Semantic Transfer at Grounded Execution in Embodied Agents](https://mlsanity.com/paper/2609.13458/) (2026)

## Abstract (arXiv)

Natural language understanding for robotics can require substantial domain- and platform-specific engineering. For example, for mobile robots to pick-and-place objects in an environment to satisfy human commands, we can specify the language humans use to issue such commands, and connect concept words like red can to physical object properties. One way to alleviate this engineering for a new domain is to enable robots in human environments to adapt dynamically---continually learning new language constructions and perceptual concepts. In this work, we present an end-to-end pipeline for translating natural language commands to discrete robot actions, and use clarification dialogs to jointly improve language parsing and concept grounding. We train and evaluate this agent in a virtual setting on Amazon Mechanical Turk, and we transfer the learned agent to a physical robot platform to demonstrate it in the real world.