# AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios

arXiv: https://arxiv.org/abs/2410.19346 · HTML: https://mlsanity.com/paper/2410.19346/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 61 references that are in the map.
- First posted on arXiv 25 Oct 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 23 of 23)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Character-LLM: A Trainable Agent for Role-Playing](https://mlsanity.com/paper/2310.10158/) (2023)
- [Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf](https://mlsanity.com/paper/2309.04658/) (2023)
- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024)
- [Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback](https://mlsanity.com/paper/2305.10142/) (2023)
- [Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models](https://mlsanity.com/paper/2305.14763/) (2023)
- [MetaAgents: Large Language Model Based Agents for Decision-Making on Teaming](https://mlsanity.com/paper/2310.06500/) (2023)
- [Unveiling the Truth and Facilitating Change: Towards Agent-based Large-scale Social Movement Simulation](https://mlsanity.com/paper/2402.16333/) (2024)
- [Can Large Language Model Agents Simulate Human Trust Behavior?](https://mlsanity.com/paper/2402.04559/) (2024)
- [Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs](https://mlsanity.com/paper/2403.05020/) (2024)
- [From Skepticism to Acceptance: Simulating the Attitude Dynamics Toward Fake News](https://mlsanity.com/paper/2403.09498/) (2024)
- [SocialBench: Sociality Evaluation of Role-Playing Conversational Agents](https://mlsanity.com/paper/2403.13679/) (2024)
- [The Wisdom of Partisan Crowds: Comparing Collective Intelligence in Humans and LLM-based Agents](https://mlsanity.com/paper/2311.09665/) (2023)
- [Towards Objectively Benchmarking Social Intelligence for Language Agents at Action Level](https://mlsanity.com/paper/2404.05337/) (2024)

## Built on by (most cited first, 30 of 40)

- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents](https://mlsanity.com/paper/2412.03563/) (2024)
- [Multi-Agent Simulator Drives Language Models for Legal Intensive Interaction](https://mlsanity.com/paper/2502.06882/) (2025)
- [Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents](https://mlsanity.com/paper/2502.13012/) (2025)
- [Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models](https://mlsanity.com/paper/2505.02847/) (2025)
- [SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agents](https://mlsanity.com/paper/2502.15538/) (2025)
- [The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies](https://mlsanity.com/paper/2509.18052/) (2025)
- [Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments](https://mlsanity.com/paper/2507.04037/) (2025)
- [Multi-agent KTO: Reinforcing Strategic Interactions of Large Language Model in Language Game](https://mlsanity.com/paper/2501.14225/) (2025)
- [PIORS: Personalized Intelligent Outpatient Reception based on Large Language Model with Multi-Agents Medical Scenario Simulation](https://mlsanity.com/paper/2411.13902/) (2024)
- [HumanLLM: Towards Personalized Understanding and Simulation of Human Nature](https://mlsanity.com/paper/2601.15793/) (2026)
- [SI-Bench: Benchmarking Social Intelligence of Large Language Models in Human-to-Human Conversations](https://mlsanity.com/paper/2510.23182/) (2025)
- [One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligence](https://mlsanity.com/paper/2602.03109/) (2026)
- [Multimodal Safety Evaluation in Generative Agent Social Simulations](https://mlsanity.com/paper/2510.07709/) (2025)
- [VIBE: Can a VLM Read the Room?](https://mlsanity.com/paper/2506.11162/) (2025)
- [Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understanding](https://mlsanity.com/paper/2508.20765/) (2025)
- [ProMediate: A Socio-cognitive framework for evaluating proactive agents in multi-party negotiation](https://mlsanity.com/paper/2510.25224/) (2025)
- [Large Language Models Hack Rewards, and Society](https://mlsanity.com/paper/2606.04075/) (2026)
- [Aligning VLM Assistants with Personalized Situated Cognition](https://mlsanity.com/paper/2506.00930/) (2025)
- [ConsintBench: Evaluating Language Models on Real-World Consumer Intent Understanding](https://mlsanity.com/paper/2510.13499/) (2025)
- [Sentipolis: Emotion-Aware Agents for Social Simulations](https://mlsanity.com/paper/2601.18027/) (2026)
- [SocialVeil: Probing Social Intelligence of Language Agents under Communication Barriers](https://mlsanity.com/paper/2602.05115/) (2026)
- [LLM Agents for Deliberative Collaboration: A Study on Joint Decision Making Under Partial Observability](https://mlsanity.com/paper/2607.06157/) (2026)
- [Auto-SLURP: A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistant](https://mlsanity.com/paper/2504.18373/) (2025)
- [Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Design](https://mlsanity.com/paper/2608.10299/) (2026)
- [SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Design](https://mlsanity.com/paper/2609.09764/) (2026)
- [PeerCoPilot: A Language Model-Powered Assistant for Behavioral Health Organizations](https://mlsanity.com/paper/2511.21721/) (2025)
- [COINBench: Moving Beyond Individual Perspectives to Collective Intent Understanding](https://mlsanity.com/paper/2603.21329/) (2026)
- [ALSO: Adversarial Online Strategy Optimization for Social Agents](https://mlsanity.com/paper/2605.15768/) (2026)
- [Psi-Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues](https://mlsanity.com/paper/2606.02754/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly leveraged to empower autonomous agents to simulate human beings in various fields of behavioral research. However, evaluating their capacity to navigate complex social interactions remains a challenge. Previous studies face limitations due to insufficient scenario diversity, complexity, and a single-perspective focus. To this end, we introduce AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios. Drawing on Dramaturgical Theory, AgentSense employs a bottom-up approach to create 1,225 diverse social scenarios constructed from extensive scripts. We evaluate LLM-driven agents through multi-turn interactions, emphasizing both goal completion and implicit reasoning. We analyze goals using ERG theory and conduct comprehensive experiments. Our findings highlight that LLMs struggle with goals in complex social scenarios, especially high-level growth needs, and even GPT-4o requires improvement in private information reasoning. Code and data are available at urlhttps://github.com/ljcleo/agentsense.