# Using Natural Language for Human-Robot Collaboration in the Real World

arXiv: https://arxiv.org/abs/2508.11759 · HTML: https://mlsanity.com/paper/2508.11759/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 38 references that are in the map.
- First posted on arXiv 15 Aug 2025.

## Builds on (most cited first, 17 of 17)

- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions](https://mlsanity.com/paper/1710.06280/) (2017)
- [Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction](https://mlsanity.com/paper/1806.03831/) (2018)
- [A Survey of Robotic Language Grounding: Tradeoffs between Symbols and Embeddings](https://mlsanity.com/paper/2405.13245/) (2024)
- [HandMeThat: Human-Robot Communication in Physical and Social Environments](https://mlsanity.com/paper/2310.03779/) (2023)
- [Improved Trust in Human-Robot Collaboration with ChatGPT](https://mlsanity.com/paper/2304.12529/) (2023)
- [Exploiting Language Models as a Source of Knowledge for Cognitive Agents](https://mlsanity.com/paper/2310.06846/) (2023)
- [VAL: Interactive Task Learning with GPT Dialog Parsing](https://mlsanity.com/paper/2310.01627/) (2023)
- [Enabling Robots to Understand Incomplete Natural Language Instructions Using Commonsense Reasoning](https://mlsanity.com/paper/1904.12907/) (2019)
- [Challenges in Human-Agent Communication](https://mlsanity.com/paper/2412.10380/) (2024)
- [Language Models as a Knowledge Source for Cognitive Agents](https://mlsanity.com/paper/2109.08270/) (2021)
- [Integrating Diverse Knowledge Sources for Online One-shot Learning of Novel Tasks](https://mlsanity.com/paper/2208.09554/) (2022)
- [Improving Language Model Prompting in Support of Semi-autonomous Task Learning](https://mlsanity.com/paper/2209.07636/) (2022)
- [Acquiring Grounded Representations of Words with Situated Interactive Instruction](https://mlsanity.com/paper/2502.20754/) (2025)

## Abstract (arXiv)

We have a vision of a day when autonomous robots can collaborate with humans as assistants in performing complex tasks in the physical world. This vision includes that the robots will have the ability to communicate with their human collaborators using language that is natural to the humans. Traditional Interactive Task Learning (ITL) systems have some of this ability, but the language they can understand is very limited. The advent of large language models (LLMs) provides an opportunity to greatly improve the language understanding of robots, yet integrating the language abilities of LLMs with robots that operate in the real physical world is a challenging problem. In this chapter we first review briefly a few commercial robot products that work closely with humans, and discuss how they could be much better collaborators with robust language abilities. We then explore how an AI system with a cognitive agent that controls a physical robot at its core, interacts with both a human and an LLM, and accumulates situational knowledge through its experiences, can be a possible approach to reach that vision. We focus on three specific challenges of having the robot understand natural language, and present a simple proof-of-concept experiment using ChatGPT for each. Finally, we discuss what it will take to turn these simple experiments into an operational system where LLM-assisted language understanding is a part of an integrated robotic assistant that uses language to collaborate with humans.