# VIBE: Can a VLM Read the Room?

arXiv: https://arxiv.org/abs/2506.11162 · HTML: https://mlsanity.com/paper/2506.11162/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 62 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Builds on (most cited first, 28 of 28)

- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Multi-Modal Hallucination Control by Visual Information Grounding](https://mlsanity.com/paper/2403.14003/) (2024)
- [An Introduction to Vision-Language Modeling](https://mlsanity.com/paper/2405.17247/) (2024)
- [M-LLM Based Video Frame Selection for Efficient Video Understanding](https://mlsanity.com/paper/2502.19680/) (2025)
- [EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributes](https://mlsanity.com/paper/2307.07961/) (2023)
- [Context Based Emotion Recognition using EMOTIC Dataset](https://mlsanity.com/paper/2003.13401/) (2020)
- [V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization](https://mlsanity.com/paper/2411.02712/) (2024)
- [PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities](https://mlsanity.com/paper/2401.07078/) (2024)
- [ALOHa: A New Measure for Hallucination in Captioning Models](https://mlsanity.com/paper/2404.02904/) (2024)
- [CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries](https://mlsanity.com/paper/2501.01282/) (2025)
- [AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios](https://mlsanity.com/paper/2410.19346/) (2024)
- [Enhancing Emotional Generation Capability of Large Language Models via Emotional Chain-of-Thought](https://mlsanity.com/paper/2401.06836/) (2024)
- [SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Models](https://mlsanity.com/paper/2312.09818/) (2023)
- [InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Context](https://mlsanity.com/paper/2406.12203/) (2024)
- [Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)](https://mlsanity.com/paper/2408.04664/) (2024)
- [VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework](https://mlsanity.com/paper/2403.09027/) (2024)
- [Does Your Vision-Language Model Get Lost in the Long Video Sampling Dilemma?](https://mlsanity.com/paper/2503.12496/) (2025)
- [Evaluating Vision-Language Models for Emotion Recognition](https://mlsanity.com/paper/2502.05660/) (2025)
- [Emotion and Intent Joint Understanding in Multimodal Conversation: A Benchmarking Dataset](https://mlsanity.com/paper/2407.02751/) (2024)
- [Face-MLLM: A Large Face Perception Model](https://mlsanity.com/paper/2410.20717/) (2024)
- [Large Vision-Language Models as Emotion Recognizers in Context Awareness](https://mlsanity.com/paper/2407.11300/) (2024)
- [Visual Prompting in LLMs for Enhancing Emotion Recognition](https://mlsanity.com/paper/2410.02244/) (2024)
- [A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues](https://mlsanity.com/paper/2305.04530/) (2023)
- [Evaluating Pragmatic Abilities of Image Captioners on A3DS](https://mlsanity.com/paper/2305.12777/) (2023)
- [Resolving References in Visually-Grounded Dialogue via Text Generation](https://mlsanity.com/paper/2309.13430/) (2023)

## Built on by (most cited first, 4 of 4)

- [Using Vision-Language Models as Proxies for Social Intelligence in Human-Robot Interaction](https://mlsanity.com/paper/2512.07177/) (2025)
- [Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?](https://mlsanity.com/paper/2512.17394/) (2025)
- [Chehre: An Emoji-Prompted Dataset to Explore Perceptual Flexibility in Video Language Models](https://mlsanity.com/paper/2606.21657/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)

## Abstract (arXiv)

Understanding human social behavior such as recognizing emotions and the social dynamics causing them is an important and challenging problem. While LLMs have made remarkable advances, they are limited to the textual domain and cannot account for the major role that non-verbal cues play in understanding social situations. Vision Language Models (VLMs) can potentially account for this gap, however their ability to make correct inferences over such social cues has received little attention. In this paper, we explore the capabilities of VLMs at social reasoning. We identify a previously overlooked limitation in VLMs: the Visual Social-Pragmatic Inference gap. To target this gap, we propose a new task for VLMs: Visual Social-Pragmatic Inference. We construct a high quality dataset to test the abilities of a VLM for this task and benchmark the performance of several VLMs on it.