mlsanityData to 7 Oct 2026

Social Caption: Evaluating Social Understanding in Multimodal Models

arXiv 2601.14569 ↗First posted 21 Jan 2026cs.CL cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Social understanding abilities are crucial for multimodal large language models (MLLMs) to interpret human social interactions. We introduce SOCIAL CAPTION, a framework grounded in interaction theory to evaluate social understanding abilities of MLLMs along three dimensions: Social Inference (SI), the ability to make accurate inferences about interactions; Holistic Social Analysis (HSA), the ability to generate comprehensive descriptions of interactions; Directed Social Analysis (DSA), the ability to generate relevant information from interactions. We analyze factors influencing model performance in social understanding, such as scale, architectural design, and spoken context. Experiments with MLLM judges demonstrate a path towards scaling automated evaluation of multimodal social understanding.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 66 references that are in the map. First posted on arXiv 21 Jan 2026.

Builds on

33 of 66 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025MR. Judge: Multimodal Reasoner as a Judge19 May
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  3. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks12 May
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  5. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark7 Feb
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  10. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  11. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  12. 2024Long Context Transfer from Language to Visioncited by 502
15 more
  1. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  2. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  3. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  4. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 320
  5. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  6. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  7. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  8. 2023COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statementscited by 34
  9. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 33
  10. 2025Social Genome: Grounded Social Reasoning Abilities of Multimodal Modelscited by 24
  11. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  12. 2025SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoningcited by 18
  13. 2025EgoNormia: Benchmarking Physical Social Norm Understandingcited by 13
  14. 2025MimeQA: Towards Socially-Intelligent Nonverbal Foundation Modelscited by 9
  15. 2025VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understandingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactionscited by 1
  2. 2026Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Social AI: A Survey on Understanding Social Interactions
  2. 2026PIVOTSBench: Evaluating Fine-Grained Interpersonal Relationship Reasoning in Multimodal Large Language Models
  3. 2025Towards Online Multi-Modal Social Interaction Understanding
  4. 2025VIBE: Can a VLM Read the Room?
  5. 2023Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark
  6. 2025VEGAS: Towards Visually Explainable and Grounded Artificial Social Intelligence
  7. 2025Interpreting Social Bias in LVLMs via Information Flow Analysis and Multi-Round Dialogue Evaluation
  8. 2026Can Agents Read the Room? Benchmarking Visual Social Intelligence in Multimodal Simulation