# When Robots Get Chatty: Grounding Multimodal Human-Robot Conversation and Collaboration

arXiv: https://arxiv.org/abs/2407.00518 · HTML: https://mlsanity.com/paper/2407.00518/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2024.

## Built on by (most cited first, 5 of 5)

- [Robots Can Multitask Too: Integrating a Memory Architecture and LLMs for Enhanced Cross-Task Robot Action Generation](https://mlsanity.com/paper/2407.13505/) (2024)
- [Towards Multimodal Social Conversations with Robots: Using Vision-Language Models](https://mlsanity.com/paper/2507.19196/) (2025)
- [Shaken, Not Stirred: A Novel Dataset for Visual Understanding of Glasses in Human-Robot Bartending Tasks](https://mlsanity.com/paper/2503.04308/) (2025)
- [Enhancing Explainability with Multimodal Context Representations for Smarter Robots](https://mlsanity.com/paper/2503.16467/) (2025)
- [Balancing long- and short-term dynamics for the modeling of saliency in videos](https://mlsanity.com/paper/2504.05913/) (2025)

## Abstract (arXiv)

We investigate the use of Large Language Models (LLMs) to equip neural robotic agents with human-like social and cognitive competencies, for the purpose of open-ended human-robot conversation and collaboration. We introduce a modular and extensible methodology for grounding an LLM with the sensory perceptions and capabilities of a physical robot, and integrate multiple deep learning models throughout the architecture in a form of system integration. The integrated models encompass various functions such as speech recognition, speech generation, open-vocabulary object detection, human pose estimation, and gesture detection, with the LLM serving as the central text-based coordinating unit. The qualitative and quantitative results demonstrate the huge potential of LLMs in providing emergent cognition and interactive language-oriented control of robots in a natural and social manner.