mlsanityData to 7 Oct 2026

AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios

arXiv 2410.19346 ↗First posted 25 Oct 2024cs.CL cs.CYcited by 37

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language models (LLMs) are increasingly leveraged to empower autonomous agents to simulate human beings in various fields of behavioral research. However, evaluating their capacity to navigate complex social interactions remains a challenge. Previous studies face limitations due to insufficient scenario diversity, complexity, and a single-perspective focus. To this end, we introduce AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios. Drawing on Dramaturgical Theory, AgentSense employs a bottom-up approach to create 1,225 diverse social scenarios constructed from extensive scripts. We evaluate LLM-driven agents through multi-turn interactions, emphasizing both goal completion and implicit reasoning. We analyze goals using ERG theory and conduct comprehensive experiments. Our findings highlight that LLMs struggle with goals in complex social scenarios, especially high-level growth needs, and even GPT-4o requires improvement in private information reasoning. Code and data are available at urlhttps://github.com/ljcleo/agentsense.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 23 of its 61 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

23 of 61 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Towards Objectively Benchmarking Social Intelligence for Language Agents at Action Level8 Apr
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society31 Mar

Other references · 21

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  8. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  9. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  10. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 335
  11. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  12. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
9 more
  1. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  2. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 161
  3. 2023MetaAgents: Large Language Model Based Agents for Decision-Making on Teamingcited by 85
  4. 2024Unveiling the Truth and Facilitating Change: Towards Agent-based Large-scale Social Movement Simulationcited by 69
  5. 2024Can Large Language Model Agents Simulate Human Trust Behavior?cited by 62
  6. 2024Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMscited by 59
  7. 2024From Skepticism to Acceptance: Simulating the Attitude Dynamics Toward Fake Newscited by 51
  8. 2024SocialBench: Sociality Evaluation of Role-Playing Conversational Agentscited by 48
  9. 2023The Wisdom of Partisan Crowds: Comparing Collective Intelligence in Humans and LLM-based Agentscited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 17; 2026: 18.

37 papers.

Sort:

By area:

  1. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  2. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  3. 2025Multi-Agent Simulator Drives Language Models for Legal Intensive Interactioncited by 22
  4. 2025Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agentscited by 22
  5. 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
  6. 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agentscited by 16
  7. 2025Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environmentscited by 13
  8. 2025The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societiescited by 12
  9. 2025Multi-agent KTO: Reinforcing Strategic Interactions of Large Language Model in Language Gamecited by 10
  10. 2024PIORS: Personalized Intelligent Outpatient Reception based on Large Language Model with Multi-Agents Medical Scenario Simulationcited by 8
  11. 2026One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligencecited by 7
  12. 2025Multimodal Safety Evaluation in Generative Agent Social Simulationscited by 5
25 more
  1. 2025SI-Bench: Benchmarking Social Intelligence of Large Language Models in Human-to-Human Conversationscited by 5
  2. 2026HumanLLM: Towards Personalized Understanding and Simulation of Human Naturecited by 5
  3. 2025VIBE: Can a VLM Read the Room?cited by 4
  4. 2025Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understandingcited by 3
  5. 2026Large Language Models Hack Rewards, and Societycited by 3
  6. 2025Aligning VLM Assistants with Personalized Situated Cognitioncited by 2
  7. 2025ConsintBench: Evaluating Language Models on Real-World Consumer Intent Understandingcited by 2
  8. 2025ProMediate: A Socio-cognitive framework for evaluating proactive agents in multi-party negotiationcited by 2
  9. 2026Sentipolis: Emotion-Aware Agents for Social Simulationscited by 2
  10. 2026SocialVeil: Probing Social Intelligence of Language Agents under Communication Barrierscited by 2
  11. 2025Auto-SLURP: A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistantcited by 1
  12. 2026Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Designcited by 1
  13. 2026SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Designcited by 1
  14. 2025PeerCoPilot: A Language Model-Powered Assistant for Behavioral Health Organizationscited by 0
  15. 2026COINBench: Moving Beyond Individual Perspectives to Collective Intent Understandingcited by 0
  16. 2026ALSO: Adversarial Online Strategy Optimization for Social Agentscited by 0
  17. 2026Psi-Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialoguescited by 0
  18. 2026Can Agents Read the Room? Benchmarking Visual Social Intelligence in Multimodal Simulationcited by 0
  19. 2026PIVOTSBench: Evaluating Fine-Grained Interpersonal Relationship Reasoning in Multimodal Large Language Modelscited by 0
  20. 2026LLM Agents for Deliberative Collaboration: A Study on Joint Decision Making Under Partial Observabilitycited by 0
  21. 2026Reproducing human biases in route choice using large language models: Toward scalable behavioral modelingcited by 0
  22. 2026IB-RL: Isolated Bilateral Reinforcement Learning for Strategic Dialogue Agentscited by 0
  23. 2026LiveSim: Simulating Environment-Shaped Users in Multi-Agent Live-Stream Ecosystemscited by 0
  24. 2026GLARE: Generative Learning via Adversarial Reward Estimation For Social Dynamics Forecastingcited by 0
  25. 2026Towards Communication-Efficient Social Intelligence in Language Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarios
  2. 2026Communicate-Predict-Act: Evaluating Social Intelligence of Agents
  3. 2025The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation
  4. 2025LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactions
  5. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents
  6. 2025SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments
  7. 2025How Social is It? A Benchmark for LLMs' Capabilities in Multi-user Multi-turn Social Agent Tasks
  8. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View