mlsanityData to 14 Oct 2026

Towards Multimodal Social Conversations with Robots: Using Vision-Language Models

arXiv 2507.19196 ↗First posted 25 Jul 2025cs.RO cs.CL cs.HCcited by 3

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large language models have given social robots the ability to autonomously engage in open-domain conversations. However, they are still missing a fundamental social skill: making use of the multiple modalities that carry social interactions. While previous work has focused on task-oriented interactions that require referencing the environment or specific phenomena in social interactions such as dialogue breakdowns, we outline the overall needs of a multimodal system for social conversations with robots. We then argue that vision-language models are able to process this wide range of visual information in a sufficiently general manner for autonomous social robots. We describe how to adapt them to this setting, which technical challenges remain, and briefly discuss evaluation practices.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 26 references that are in the map. First posted on arXiv 25 Jul 2025.

Builds on

7 of 26 references.

References · 7

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  5. 2020Defining and Quantifying Conversation Quality in Spontaneous Interactionscited by 8
  6. 2024When Robots Get Chatty: Grounding Multimodal Human-Robot Conversation and Collaborationcited by 5
  7. 2023I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robotscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026A Framework for Low-Latency, LLM-driven Multimodal Interaction on the Pepper Robotcited by 1
  2. 2026Human-Inspired Context-Selective Multimodal Memory for Social Robotscited by 0
  3. 2026Hybrid Attention Estimation Pipeline for Adaptive HRI Using an Expressive Robotic Headcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Conversation is the Command: Interacting with Real-World Autonomous Robot Through Natural Language
  2. 2026The World According to a Social Robot -- Augmenting Human-Robot Dialogue With Vision Language Models
  3. 2024Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces
  4. 2024Multimodal Human-Autonomous Agents Interaction Using Pre-Trained Language and Visual Foundation Models
  5. 2026ARIS: Agentic and Relationship Intelligence System for Social Robots
  6. 2026A Modern System Recipe for Situated Embodied Human-Robot Conversation with Real-Time Multimodal LLMs and Tool-Calling
  7. 2026Lightweight Visual Reasoning for Socially-Aware Robots
  8. 2025Using Vision-Language Models as Proxies for Social Intelligence in Human-Robot Interaction