# Challenges in Grounding Language in the Real World

arXiv: https://arxiv.org/abs/2506.17375 · HTML: https://mlsanity.com/paper/2506.17375/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 22 references that are in the map.
- First posted on arXiv 20 Jun 2025.

## Builds on (most cited first, 7 of 7)

- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction](https://mlsanity.com/paper/1806.03831/) (2018)
- [A Survey of Robotic Language Grounding: Tradeoffs between Symbols and Embeddings](https://mlsanity.com/paper/2405.13245/) (2024)
- [VAL: Interactive Task Learning with GPT Dialog Parsing](https://mlsanity.com/paper/2310.01627/) (2023)
- [Enabling Robots to Understand Incomplete Natural Language Instructions Using Commonsense Reasoning](https://mlsanity.com/paper/1904.12907/) (2019)
- [Integrating Diverse Knowledge Sources for Online One-shot Learning of Novel Tasks](https://mlsanity.com/paper/2208.09554/) (2022)

## Abstract (arXiv)

A long-term goal of Artificial Intelligence is to build a language understanding system that allows a human to collaborate with a physical robot using language that is natural to the human. In this paper we highlight some of the challenges in doing this, and propose a solution that integrates the abilities of a cognitive agent capable of interactive task learning in a physical robot with the linguistic abilities of a large language model. We also point the way to an initial implementation of this approach.