mlsanityData to 7 Oct 2026

Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs

arXiv 2403.05020 ↗First posted 8 Mar 2024cs.CL cs.AIcited by 59

Not on the map. Why?

Cited by 26 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in large language models (LLM) have enabled richer social simulations, allowing for the study of various social phenomena. However, most recent work has used a more omniscient perspective on these simulations (e.g., single LLM to generate all interlocutors), which is fundamentally at odds with the non-omniscient, information asymmetric interactions that involve humans and AI agents in the real world. To examine these differences, we develop an evaluation framework to simulate social interactions with LLMs in various settings (omniscient, non-omniscient). Our experiments show that LLMs perform better in unrealistic, omniscient simulation settings but struggle in ones that more accurately reflect real-world conditions with information asymmetry. Our findings indicate that addressing information asymmetry remains a fundamental challenge for LLM-based agents.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 30 of its 69 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

30 of 69 references.

References · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  3. 2018Model Cards for Model Reportingcited by 1,068
  4. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  5. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
  6. 2023An Overview of Catastrophic AI Riskscited by 214
  7. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  8. 2022Talking About Large Language Modelscited by 174
  9. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
  10. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 162
  11. 2018Decoupling Strategy and Generation in Negotiation Dialoguescited by 156
  12. 2020Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skillscited by 130
18 more
  1. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
  2. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 95
  3. 2023Humanoid Agents: Platform for Simulating Human-like Generative Agentscited by 73
  4. 2023PLACES: Prompting Language Models for Social Conversation Synthesiscited by 72
  5. 2017Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddingscited by 59
  6. 2022The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMscited by 57
  7. 2022Inferring Rewards from Language in Contextcited by 50
  8. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulationcited by 46
  9. 2023Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversationscited by 38
  10. 2024Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talkcited by 34
  11. 2021NOPE: A Corpus of Naturally-Occurring Presuppositions in Englishcited by 23
  12. 2024Under the Surface: Tracking the Artifactuality of LLM-Generated Datacited by 23
  13. 2021Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identitycited by 22
  14. 2023Reproducibility in NLP: What Have We Learned from the Checklist?cited by 18
  15. 2023Inferring the Goals of Communicating Agents from Actions and Instructionscited by 17
  16. 2023Evaluating statistical language models as pragmatic reasonerscited by 14
  17. 2018The division of labor in communication: Speakers help listeners account for asymmetries in visual perspectivecited by 6
  18. 2022Resolving Indirect Referring Expressions for Entity Selectioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 23; 2026: 13.

59 papers.

Sort:

By area:

  1. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  2. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  3. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  4. 2024On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agentscited by 92
  5. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  6. 2024Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principlescited by 72
  7. 2024Cactus: Towards Psychological Counseling Conversations using Cognitive Behavioral Theorycited by 60
  8. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  9. 2024Large Language Models Assume People are More Rational than We Really arecited by 49
  10. 2024GenSim: A General Social Simulation Platform with Large Language Model based Agentscited by 48
  11. 2024AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarioscited by 37
  12. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33
38 more
  1. 2025Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challengescited by 31
  2. 2024AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agentscited by 28
  3. 2024Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?cited by 26
  4. 2024InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Contextcited by 25
  5. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 23
  6. 2024BattleAgent: Multi-modal Dynamic Emulation on Historical Battles to Complement Historical Analysiscited by 20
  7. 2024LLM-Generated Black-box Explanations Can Be Adversarially Helpfulcited by 20
  8. 2024Do Large Language Models Align with Core Mental Health Counseling Competencies?cited by 19
  9. 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agentscited by 16
  10. 2025Cultural Learning-Based Culture Adaptation of Language Modelscited by 16
  11. 2024Autonomous Agents for Collaborative Task under Information Asymmetrycited by 15
  12. 2024Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialoguecited by 15
  13. 2025Can A Society of Generative Agents Simulate Human Behavior and Inform Public Health Policy? A Case Study on Vaccine Hesitancycited by 15
  14. 2025Social World Modelscited by 14
  15. 2024Social Skill Training with Large Language Modelscited by 13
  16. 2025Generative Value Conflicts Reveal LLM Prioritiescited by 13
  17. 2025The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societiescited by 12
  18. 2024Cohesive Conversations: Enhancing Authenticity in Multi-Agent Simulated Dialoguescited by 11
  19. 2025SOTOPIA-S4: a user-friendly system for flexible, customizable, and large-scale social simulationcited by 10
  20. 2024A Large-Scale Simulation on Large Language Models for Decision-Making in Political Sciencecited by 9
  21. 2025Beyond Static Responses: Multi-Agent LLM Systems as a New Paradigm for Social Science Researchcited by 9
  22. 2025From the Fluency Fallacy to the Micro-to-Macro Validity Gap: Opportunities and Pitfalls of LLMs in Social Simulationcited by 8
  23. 2026MirrorBench: A Benchmark to Evaluate Conversational User-Proxy Agents for Human-Likenesscited by 7
  24. 2024DEMO: Reframing Dialogue Interaction with Fine-grained Element Modelingcited by 6
  25. 2025Bottom-Up Synthesis of Knowledge-Grounded Task-Oriented Dialogues with Iteratively Self-Refined Promptscited by 5
  26. 2025Time to Talk: LLM Agents for Asynchronous Group Communication in Mafia Gamescited by 5
  27. 2025LLM Agents Display Human Biases but Exhibit Distinct Learning Patternscited by 3
  28. 2025How Social is It? A Benchmark for LLMs' Capabilities in Multi-user Multi-turn Social Agent Taskscited by 3
  29. 2025Implicit Behavioral Alignment of Language Agents in High-Stakes Crowd Simulationscited by 3
  30. 2025You need to MIMIC to get FAME: Solving Meeting Transcript Scarcity with a Multi-Agent Conversationscited by 2
  31. 2025What Makes LLM Agent Simulations Useful for Policy Practice? An Iterative Design Study in Emergency Preparednesscited by 2
  32. 2025Virus Infection Attack on LLMs: Your Poisoning Can Spread "VIA" Synthetic Datacited by 2
  33. 2025Communication and Verification in LLM Agents towards Collaboration under Information Asymmetrycited by 2
  34. 2026Auditing Support Strategies in LLMs through Grounded Multi-Turn Social Simulationcited by 2
  35. 2025The Imperfect Learner: Incorporating Developmental Trajectories in Memory-based Student Simulationcited by 1
  36. 2026Distributed Partial Information Puzzles: Examining Common Ground Construction Under Epistemic Asymmetrycited by 1
  37. 2026CCD-CBT: Multi-Agent Therapeutic Interaction for CBT Guided by Cognitive Conceptualization Diagramcited by 1
  38. 2026Diversity Without Fidelity: A Solver-Sampler Mismatch in Multi-Agent LLM Negotiation Simulationcited by 1

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLM-Augmented Agent-Based Modelling for Social Simulations: Challenges and Opportunities
  2. 2024Systematic Biases in LLM Simulations of Debates
  3. 2026AI Agents Alone Are Not (Yet) Sufficient for Social Simulation
  4. 2024Sense and Sensitivity: Evaluating the simulation of social dynamics via Large Language Models
  5. 2024Shall We Team Up: Exploring Spontaneous Cooperation of Competing LLM Agents
  6. 2026Should LLM Agents Decide in Social Simulations? Comparing Finite-State and LLM-Based Decision Policies
  7. 2025Don't Trust Generative Agents to Mimic Communication on Social Networks Unless You Benchmarked their Empirical Realism
  8. 2025Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulations