mlsanityData to 14 Oct 2026

Social Genome: Grounded Social Reasoning Abilities of Multimodal Models

arXiv 2502.15109 ↗First posted 21 Feb 2025cs.CL cs.LGcited by 26

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Social reasoning abilities are crucial for AI systems to effectively interpret and respond to multimodal human communication and interaction within social contexts. We introduce SOCIAL GENOME, the first benchmark for fine-grained, grounded social reasoning abilities of multimodal models. SOCIAL GENOME contains 272 videos of interactions and 1,486 human-annotated reasoning traces related to inferences about these interactions. These traces contain 5,777 reasoning steps that reference evidence from visual cues, verbal cues, vocal cues, and external knowledge (contextual knowledge external to videos). SOCIAL GENOME is also the first modeling challenge to study external knowledge in social reasoning. SOCIAL GENOME computes metrics to holistically evaluate semantic and structural qualities of model-generated social reasoning traces. We demonstrate the utility of SOCIAL GENOME through experiments with state-of-the-art models, identifying performance gaps and opportunities for future research to improve the grounded social reasoning abilities of multimodal models.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 20 of its 68 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

20 of 68 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024An Incomplete Loop: Deductive, Inductive, and Abductive Reasoning in Language Models3 Apr

Other references · 19

  1. 2024GPT-4o System Cardcited by 4,588
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  4. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  5. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  6. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  7. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  8. 2024Long Context Transfer from Language to Visioncited by 515
  9. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  10. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  11. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  12. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 112
7 more
  1. 2020Learning to Explain: Datasets and Models for Identifying Valid Reasoning Chains in Multihop Question-Answeringcited by 49
  2. 2024Through the Theory of Mind's Eye: Reading Minds with Multimodal Video Large Language Modelscited by 9
  3. 2024Larger Language Models Don't Care How You Think: Why Chain-of-Thought Prompting Fails in Subjective Taskscited by 8
  4. 2023DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence Understandingcited by 5
  5. 2024LLMs Meet Long Video: Advancing Long Video Question Answering with An Interactive Visual Adapter in LLMscited by 5
  6. 2023Digital Socrates: Evaluating LLMs through Explanation Critiquescited by 4
  7. 2023Evaluating In-Context Learning of Libraries for Code Generationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 13.

26 papers.

Sort:

By area:

  1. 2025SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoningcited by 18
  2. 2025MOMENTS: A Comprehensive Multimodal Benchmark for Theory of Mindcited by 13
  3. 2026SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Modelscited by 13
  4. 2025Humanizing Machines: Rethinking LLM Anthropomorphism Through a Multi-Level Framework of Designcited by 11
  5. 2025MimeQA: Towards Socially-Intelligent Nonverbal Foundation Modelscited by 9
  6. 2025Multi-speaker Attention Alignment for Multimodal Social Interactioncited by 9
  7. 2025PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehuntscited by 8
  8. 2026Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RLcited by 8
  9. 2025VideoNorms: Benchmarking Cultural Awareness of Video Language Modelscited by 4
  10. 2025Cognitive Chain-of-Thought (CoCoT): Structured Multimodal Reasoning about Social Situationscited by 3
  11. 2025Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understandingcited by 3
  12. 2025Social Human Robot Embodied Conversation (SHREC) Dataset: Benchmarking Foundational Models' Social Reasoningcited by 2
14 more
  1. 2026Social Caption: Evaluating Social Understanding in Multimodal Modelscited by 2
  2. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Modelscited by 1
  3. 2025Mind the Motions: Benchmarking Theory-of-Mind in Everyday Body Languagecited by 1
  4. 2026GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactionscited by 1
  5. 2026Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025cited by 1
  6. 2025Spot The Ball: A Benchmark for Visual Social Inferencecited by 0
  7. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
  8. 2026Act2See: Emergent Active Visual Perception for Video Reasoningcited by 0
  9. 2026COSI-Lab: Conference Living Lab for Modeling Multi-Perspective Multimodal Social Intentioncited by 0
  10. 2026MoCA: Implicit Social Context Analysiscited by 0
  11. 2026SocialReasonBench: A Video-QA Benchmark for Social Reasoning with Counterfactual Narrative Videoscited by 0
  12. 2026AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalizationcited by 0
  13. 2026Reinforcement Learning with Comparative Evidence for Social Intelligencecited by 0
  14. 2026GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Social AI: A Survey on Understanding Social Interactions
  2. 2025SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments
  3. 2025SVBench: Evaluation of Video Generation Models on Social Reasoning
  4. 2023Understanding Social Reasoning in Language Models with Language Models
  5. 2025SI-Bench: Benchmarking Social Intelligence of Large Language Models in Human-to-Human Conversations
  6. 2026MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings
  7. 2026SCRuB: Social Concept Reasoning under Rubric-Based Evaluation
  8. 2025R^3-VQA: "Read the Room" by Video Social Reasoning