mlsanityData to 14 Oct 2026

VIBE: Can a VLM Read the Room?

arXiv 2506.11162 ↗First posted 11 Jun 2025cs.CV cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Understanding human social behavior such as recognizing emotions and the social dynamics causing them is an important and challenging problem. While LLMs have made remarkable advances, they are limited to the textual domain and cannot account for the major role that non-verbal cues play in understanding social situations. Vision Language Models (VLMs) can potentially account for this gap, however their ability to make correct inferences over such social cues has received little attention. In this paper, we explore the capabilities of VLMs at social reasoning. We identify a previously overlooked limitation in VLMs: the Visual Social-Pragmatic Inference gap. To target this gap, we propose a new task for VLMs: Visual Social-Pragmatic Inference. We construct a high quality dataset to test the abilities of a VLM for this task and benchmark the performance of several VLMs on it.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 28 of its 62 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

28 of 62 references.

References · 28

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  3. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  4. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  5. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
  6. 2024An Introduction to Vision-Language Modelingcited by 178
  7. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 88
  8. 2023EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributescited by 82
  9. 2020Context Based Emotion Recognition using EMOTIC Datasetcited by 73
  10. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimizationcited by 67
  11. 2024PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilitiescited by 59
  12. 2024ALOHa: A New Measure for Hallucination in Captioning Modelscited by 45
16 more
  1. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countriescited by 42
  2. 2024AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarioscited by 40
  3. 2024Enhancing Emotional Generation Capability of Large Language Models via Emotional Chain-of-Thoughtcited by 34
  4. 2023SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Modelscited by 28
  5. 2024InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Contextcited by 26
  6. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)cited by 23
  7. 2024VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Frameworkcited by 20
  8. 2025Does Your Vision-Language Model Get Lost in the Long Video Sampling Dilemma?cited by 20
  9. 2025Evaluating Vision-Language Models for Emotion Recognitioncited by 19
  10. 2024Emotion and Intent Joint Understanding in Multimodal Conversation: A Benchmarking Datasetcited by 15
  11. 2024Face-MLLM: A Large Face Perception Modelcited by 13
  12. 2024Large Vision-Language Models as Emotion Recognizers in Context Awarenesscited by 12
  13. 2024Visual Prompting in LLMs for Enhancing Emotion Recognitioncited by 12
  14. 2023A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Cluescited by 11
  15. 2023Evaluating Pragmatic Abilities of Image Captioners on A3DScited by 5
  16. 2023Resolving References in Visually-Grounded Dialogue via Text Generationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Using Vision-Language Models as Proxies for Social Intelligence in Human-Robot Interactioncited by 1
  2. 2025Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?cited by 1
  3. 2026Chehre: An Emoji-Prompted Dataset to Explore Perceptual Flexibility in Video Language Modelscited by 0
  4. 2026Towards High-Level Semantic Intelligencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models
  2. 2025R^3-VQA: "Read the Room" by Video Social Reasoning
  3. 2026Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models
  4. 2025SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigation
  5. 2026STARS: From Spatiotemporal Dynamics to Social Representations in Human-Robot Interaction
  6. 2025SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoning
  7. 2025VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models
  8. 2026Investigating Relational Reasoning in VLMs