mlsanityData to 7 Oct 2026

Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions

arXiv 2404.11023 ↗First posted 17 Apr 2024cs.HC cs.CL cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Building socially-intelligent AI agents (Social-AI) is a multidisciplinary, multimodal research goal that involves creating agents that can sense, perceive, reason about, learn from, and respond to affect, behavior, and cognition of other agents (human or artificial). Progress towards Social-AI has accelerated in the past decade across several computing communities, including natural language processing, machine learning, robotics, human-machine interaction, computer vision, and speech. Natural language processing, in particular, has been prominent in Social-AI research, as language plays a key role in constructing the social world. In this position paper, we identify a set of underlying technical challenges and open questions for researchers across computing communities to advance Social-AI. We anchor our discussion in the context of social intelligence concepts and prior progress in Social-AI research.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 61 of its 208 references that are in the map. First posted on arXiv 17 Apr 2024.

Builds on

61 of 208 references.

References · 61

  1. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  3. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  4. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  5. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  6. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  7. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  8. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  9. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  10. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  11. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  12. 2019Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networkscited by 228
38 more
  1. 2023Stable Bias: Analyzing Societal Representations in Diffusion Modelscited by 225
  2. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  3. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  4. 2018DialogueRNN: An Attentive RNN for Emotion Detection in Conversationscited by 216
  5. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  6. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 162
  7. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 161
  8. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  9. 2018GaitSet: Regarding Gait as a Set for Cross-View Gait Recognitioncited by 137
  10. 2022Power to the People? Opportunities and Challenges for Participatory AIcited by 137
  11. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 130
  12. 2021Toward a Perspectivist Turn in Ground Truthing for Predictive Computingcited by 129
  13. 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 125
  14. 2014Label Distribution Learningcited by 122
  15. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 112
  16. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
  17. 2023The Semantic Scholar Open Data Platformcited by 109
  18. 2020The Emerging Trends of Multi-Label Learningcited by 99
  19. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  20. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 91
  21. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 84
  22. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 81
  23. 2024RegionGPT: Towards Region Understanding Vision Language Modelcited by 71
  24. 2019SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wildcited by 62
  25. 2021Unmasking the Mask -- Evaluating Social Biases in Masked Language Modelscited by 60
  26. 2023NormBank: A Knowledge Bank of Situational Social Normscited by 60
  27. 2023Learning Visual Representations via Language-Guided Samplingcited by 32
  28. 2019Theory of Minds: Understanding Behavior in Groups Through Inverse Planningcited by 30
  29. 2022The One Where They Reconstructed 3D Humans and Environments in TV Showscited by 30
  30. 2018Personalized Machine Learning for Robot Perception of Affect and Engagement in Autism Therapycited by 29
  31. 2020Too many cooks: Bayesian inference for coordinating multi-agent collaborationcited by 29
  32. 2021Envisioning Communities: A Participatory Approach Towards AI for Social Goodcited by 29
  33. 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25
  34. 2023The SocialAI School: Insights from Developmental Psychology Towards Artificial Socio-Cultural Agentscited by 24
  35. 2021Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biasescited by 21
  36. 2024Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representationscited by 18
  37. 2023Deliberating with AI: Improving Decision-Making for the Future through Participatory AI Design and Stakeholder Deliberationcited by 17
  38. 2023Language-guided Human Motion Synthesis with Atomic Actionscited by 17

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 15; 2026: 3.

21 papers.

Sort:

By area:

  1. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24
  2. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  3. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  4. 2025Sotopia-RL: Reward Design for Social Intelligencecited by 20
  5. 2024A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Educationcited by 16
  6. 2025Towards Online Multi-Modal Social Interaction Understandingcited by 12
  7. 2025MimeQA: Towards Socially-Intelligent Nonverbal Foundation Modelscited by 9
  8. 2025Multi-speaker Attention Alignment for Multimodal Social Interactioncited by 8
  9. 2025LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactionscited by 7
  10. 2025OpenFace 3.0: A Lightweight Multitask System for Comprehensive Facial Behavior Analysiscited by 5
  11. 2025Human Behavior Atlas: Benchmarking Unified Psychological and Social Behavior Understandingcited by 5
  12. 2025Cognitive Chain-of-Thought (CoCoT): Structured Multimodal Reasoning about Social Situationscited by 3
9 more
  1. 2025Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understandingcited by 3
  2. 2025MedSimAI: Simulation and Formative Feedback Generation to Enhance Deliberate Practice in Medical Educationcited by 2
  3. 2025Reliable agent engineering should integrate machine-compatible organizational principlescited by 2
  4. 2026SocialDirector: Training-Free Social Interaction Control for Multi-Person Video Generationcited by 2
  5. 2025Social Human Robot Embodied Conversation (SHREC) Dataset: Benchmarking Foundational Models' Social Reasoningcited by 1
  6. 2025SocialFusion: Addressing Social Degradation in Pre-trained Vision-Language Modelscited by 1
  7. 2025Spot The Ball: A Benchmark for Visual Social Inferencecited by 0
  8. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
  9. 2026Watching Movies Like a Human: Egocentric Emotion Understanding for Embodied Companionscited by 0