mlsanityData to 14 Oct 2026

Using Vision-Language Models as Proxies for Social Intelligence in Human-Robot Interaction

arXiv 2512.07177 ↗First posted 8 Dec 2025cs.ROcited by 1

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Robots operating in everyday environments must often decide when and whether to engage with people, yet such decisions often hinge on subtle nonverbal cues that unfold over time and are difficult to model explicitly. Drawing on a five-day Wizard-of-Oz deployment of a mobile service robot in a university cafe, we analyze how people signal interaction readiness through nonverbal behaviors and how expert wizards use these cues to guide engagement. Motivated by these observations, we propose a two-stage pipeline in which lightweight perceptual detectors (gaze shifts and proxemics) are used to selectively trigger heavier video-based vision-language model (VLM) queries at socially meaningful moments. We evaluate this pipeline on replayed field interactions and compare two prompting strategies. Our findings suggest that selectively using VLMs as proxies for social reasoning enables socially responsive robot behavior, allowing robots to act appropriately by attending to the cues people naturally provide in real-world interactions.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 9 of its 38 references that are in the map. First posted on arXiv 8 Dec 2025.

Builds on

9 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Real-World Robot Applications of Foundation Models: A Review8 Feb
  2. 2023Physically Grounded Vision-Language Models for Robotic Manipulation5 Sep

Other references · 7

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  2. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  3. 2019Unmasking Clever Hans Predictors and Assessing What Machines Really Learncited by 365
  4. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  5. 2022ViTPose++: Vision Transformer for Generic Body Pose Estimationcited by 65
  6. 2025VIBE: Can a VLM Read the Room?cited by 4
  7. 2025Signaling Human Intentions to Service Robots: Understanding the Use of Social Cues during In-Person Conversationscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalitiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Belief Without Behavior: Measuring the Translation of Theory of Mind into Coordinated Social Action in Vision-Language Models
  2. 2024Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces
  3. 2026Lightweight Visual Reasoning for Socially-Aware Robots
  4. 2025Towards Multimodal Social Conversations with Robots: Using Vision-Language Models
  5. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactions
  6. 2024Are Large Language Models Aligned with People's Social Intuitions for Human-Robot Interactions?
  7. 2026From Social Reasoning to Embodied Interaction: An Agentic Framework for Social Robots
  8. 2024Towards Social AI: A Survey on Understanding Social Interactions