mlsanityData to 14 Oct 2026

The World According to a Social Robot -- Augmenting Human-Robot Dialogue With Vision Language Models

arXiv 2607.16318 ↗First posted 15 Jul 2026cs.HC cs.ROcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision Language Models (VLMs) enable robots to visually perceive their environment as well as the actions and characteristics of their conversation partner or humans in collaboration. Especially for social robots deployed in everyday settings and for uncomplicated, natural use, it is essential that the robot has an understanding of situations that is appropriate to human customs. This paper presents initial experiences with the application of a Mistral AI language model with a Pepper robot for Human-Robot Interaction (HRI) in dialogue, as well as an investigation of the effects of additional visual information on response time in different models. The results show that incorporating visual information adds context to the dialogue with only a moderate increase in response time, enabling both the robot and the human to take into account unspoken elements of the situation. Furthermore, using an LLM hosted in Europe offers a solution that complies with European data protection regulations and can therefore facilitate real-life applications more easily.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 8 of its 16 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

8 of 16 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactions15 Feb
  2. 2024TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models19 Dec
  3. 2024VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Models30 Mar
  4. 2023Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviors10 Mar

Other references · 4

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 68
  3. 2025SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigationcited by 10
  4. 2023I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robotscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Multimodal Social Conversations with Robots: Using Vision-Language Models
  2. 2024The Conversation is the Command: Interacting with Real-World Autonomous Robot Through Natural Language
  3. 2025Agreeing to Interact in Human-Robot Interaction using Large Language Models and Vision Language Models
  4. 2026Lightweight Visual Reasoning for Socially-Aware Robots
  5. 2024Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces
  6. 2024Multimodal Human-Autonomous Agents Interaction Using Pre-Trained Language and Visual Foundation Models
  7. 2023HuBo-VLM: Unified Vision-Language Model designed for HUman roBOt interaction tasks
  8. 2025Using Vision-Language Models as Proxies for Social Intelligence in Human-Robot Interaction