# Towards Multimodal Social Conversations with Robots: Using Vision-Language Models

arXiv: https://arxiv.org/abs/2507.19196 · HTML: https://mlsanity.com/paper/2507.19196/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 7 of its 26 references that are in the map.
- First posted on arXiv 25 Jul 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 7 of 7)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Defining and Quantifying Conversation Quality in Spontaneous Interactions](https://mlsanity.com/paper/2009.12842/) (2020)
- [When Robots Get Chatty: Grounding Multimodal Human-Robot Conversation and Collaboration](https://mlsanity.com/paper/2407.00518/) (2024)
- [I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots](https://mlsanity.com/paper/2311.08957/) (2023)

## Built on by (most cited first, 3 of 3)

- [A Framework for Low-Latency, LLM-driven Multimodal Interaction on the Pepper Robot](https://mlsanity.com/paper/2603.21013/) (2026)
- [Human-Inspired Context-Selective Multimodal Memory for Social Robots](https://mlsanity.com/paper/2604.12081/) (2026)
- [Hybrid Attention Estimation Pipeline for Adaptive HRI Using an Expressive Robotic Head](https://mlsanity.com/paper/2608.00284/) (2026)

## Abstract (arXiv)

Large language models have given social robots the ability to autonomously engage in open-domain conversations. However, they are still missing a fundamental social skill: making use of the multiple modalities that carry social interactions. While previous work has focused on task-oriented interactions that require referencing the environment or specific phenomena in social interactions such as dialogue breakdowns, we outline the overall needs of a multimodal system for social conversations with robots. We then argue that vision-language models are able to process this wide range of visual information in a sufficiently general manner for autonomous social robots. We describe how to adapt them to this setting, which technical challenges remain, and briefly discuss evaluation practices.