# The Conversation is the Command: Interacting with Real-World Autonomous Robot Through Natural Language

arXiv: https://arxiv.org/abs/2401.11838 · HTML: https://mlsanity.com/paper/2401.11838/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jan 2024.

## Built on by (most cited first, 4 of 4)

- [EMPOWER: Embodied Multi-role Open-vocabulary Planning with Online Grounding and Execution](https://mlsanity.com/paper/2408.17379/) (2024)
- [Integrating Large Language Models for UAV Control in Simulated Environments: A Modular Interaction Approach](https://mlsanity.com/paper/2410.17602/) (2024)
- [Multimodal Human-Autonomous Agents Interaction Using Pre-Trained Language and Visual Foundation Models](https://mlsanity.com/paper/2403.12273/) (2024)
- [SIL: Symbiotic Interactive Learning for Language-Conditioned Human-Agent Co-Adaptation](https://mlsanity.com/paper/2511.05203/) (2025)

## Abstract (arXiv)

In recent years, autonomous agents have surged in real-world environments such as our homes, offices, and public spaces. However, natural human-robot interaction remains a key challenge. In this paper, we introduce an approach that synergistically exploits the capabilities of large language models (LLMs) and multimodal vision-language models (VLMs) to enable humans to interact naturally with autonomous robots through conversational dialogue. We leveraged the LLMs to decode the high-level natural language instructions from humans and abstract them into precise robot actionable commands or queries. Further, we utilised the VLMs to provide a visual and semantic understanding of the robot's task environment. Our results with 99.13% command recognition accuracy and 97.96% commands execution success show that our approach can enhance human-robot interaction in real-world applications. The video demonstrations of this paper can be found at https://osf.io/wzyf6 and the code is available at our GitHub repository (https://github.com/LinusNEP/TCC_IRoNL.git).