# Enabling Robots to Understand Incomplete Natural Language Instructions Using Commonsense Reasoning

arXiv: https://arxiv.org/abs/1904.12907 · HTML: https://mlsanity.com/paper/1904.12907/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2019.

## Built on by (most cited first, 13 of 13)

- [Few-shot Subgoal Planning with Language Models](https://mlsanity.com/paper/2205.14288/) (2022)
- [Generating Executable Action Plans with Environmentally-Aware Language Models](https://mlsanity.com/paper/2210.04964/) (2022)
- [Phrase-Based Affordance Detection via Cyclic Bilateral Interaction](https://mlsanity.com/paper/2202.12076/) (2022)
- [Talk-to-Resolve: Combining scene understanding and spatial dialogue to resolve granular task ambiguity for a collocated robot](https://mlsanity.com/paper/2111.11099/) (2021)
- [Learning Visual-Audio Representations for Voice-Controlled Robots](https://mlsanity.com/paper/2109.02823/) (2021)
- [DoRO: Disambiguation of referred object for embodied agents](https://mlsanity.com/paper/2207.14205/) (2022)
- [Integrating AI Planning with Natural Language Processing: A Combination of Explicit and Tacit Knowledge](https://mlsanity.com/paper/2202.07138/) (2022)
- [Natural Language Instructions for Intuitive Human Interaction with Robotic Assistants in Field Construction Work](https://mlsanity.com/paper/2307.04195/) (2023)
- [CANVAS: Commonsense-Aware Navigation System for Intuitive Human-Robot Interaction](https://mlsanity.com/paper/2410.01273/) (2024)
- [3D Instruction Ambiguity Detection](https://mlsanity.com/paper/2601.05991/) (2026)
- [RAIDER: Tool-Equipped Large Language Model Agent for Robotic Action Issue Detection, Explanation and Recovery](https://mlsanity.com/paper/2503.17703/) (2025)
- [Challenges in Grounding Language in the Real World](https://mlsanity.com/paper/2506.17375/) (2025)
- [Using Natural Language for Human-Robot Collaboration in the Real World](https://mlsanity.com/paper/2508.11759/) (2025)

## Abstract (arXiv)

Enabling robots to understand instructions provided via spoken natural language would facilitate interaction between robots and people in a variety of settings in homes and workplaces. However, natural language instructions are often missing information that would be obvious to a human based on environmental context and common sense, and hence does not need to be explicitly stated. In this paper, we introduce Language-Model-based Commonsense Reasoning (LMCR), a new method which enables a robot to listen to a natural language instruction from a human, observe the environment around it, and automatically fill in information missing from the instruction using environmental context and a new commonsense reasoning approach. Our approach first converts an instruction provided as unconstrained natural language into a form that a robot can understand by parsing it into verb frames. Our approach then fills in missing information in the instruction by observing objects in its vicinity and leveraging commonsense reasoning. To learn commonsense reasoning automatically, our approach distills knowledge from large unstructured textual corpora by training a language model. Our results show the feasibility of a robot learning commonsense knowledge automatically from web-based textual corpora, and the power of learned commonsense reasoning models in enabling a robot to autonomously perform tasks based on incomplete natural language instructions.