# Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs

arXiv: https://arxiv.org/abs/2403.05020 · HTML: https://mlsanity.com/paper/2403.05020/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 69 references that are in the map.
- First posted on arXiv 8 Mar 2024.

## Builds on (most cited first, 30 of 30)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://mlsanity.com/paper/2305.19118/) (2023)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [An Overview of Catastrophic AI Risks](https://mlsanity.com/paper/2306.12001/) (2023)
- [Do LLMs exhibit human-like response biases? A case study in survey design](https://mlsanity.com/paper/2311.04076/) (2023)
- [Talking About Large Language Models](https://mlsanity.com/paper/2212.03551/) (2022)
- [FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions](https://mlsanity.com/paper/2310.15421/) (2023)
- [SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization](https://mlsanity.com/paper/2212.10465/) (2022)
- [Decoupling Strategy and Generation in Negotiation Dialogues](https://mlsanity.com/paper/1808.09637/) (2018)
- [Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills](https://mlsanity.com/paper/2004.08449/) (2020)
- [From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought](https://mlsanity.com/paper/2306.12672/) (2023)
- [Challenges in Automated Debiasing for Toxic Language Detection](https://mlsanity.com/paper/2102.00086/) (2021)
- [Humanoid Agents: Platform for Simulating Human-like Generative Agents](https://mlsanity.com/paper/2310.05418/) (2023)
- [PLACES: Prompting Language Models for Social Conversation Synthesis](https://mlsanity.com/paper/2302.03269/) (2023)
- [Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings](https://mlsanity.com/paper/1704.07130/) (2017)
- [The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs](https://mlsanity.com/paper/2210.14986/) (2022)
- [Inferring Rewards from Language in Context](https://mlsanity.com/paper/2204.02515/) (2022)
- [Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation](https://mlsanity.com/paper/2402.05699/) (2024)
- [Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations](https://mlsanity.com/paper/2311.05584/) (2023)
- [Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk](https://mlsanity.com/paper/2401.05033/) (2024)
- [Under the Surface: Tracking the Artifactuality of LLM-Generated Data](https://mlsanity.com/paper/2401.14698/) (2024)
- [NOPE: A Corpus of Naturally-Occurring Presuppositions in English](https://mlsanity.com/paper/2109.06987/) (2021)
- [Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identity](https://mlsanity.com/paper/2112.05843/) (2021)
- [Reproducibility in NLP: What Have We Learned from the Checklist?](https://mlsanity.com/paper/2306.09562/) (2023)
- [Inferring the Goals of Communicating Agents from Actions and Instructions](https://mlsanity.com/paper/2306.16207/) (2023)
- [Evaluating statistical language models as pragmatic reasoners](https://mlsanity.com/paper/2305.01020/) (2023)
- [The division of labor in communication: Speakers help listeners account for asymmetries in visual perspective](https://mlsanity.com/paper/1807.09000/) (2018)
- [Resolving Indirect Referring Expressions for Entity Selection](https://mlsanity.com/paper/2212.10933/) (2022)

## Built on by (most cited first, 30 of 59)

- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024)
- [Best Practices and Lessons Learned on Synthetic Data](https://mlsanity.com/paper/2404.07503/) (2024)
- [On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents](https://mlsanity.com/paper/2408.00989/) (2024)
- [From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents](https://mlsanity.com/paper/2412.03563/) (2024)
- [Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles](https://mlsanity.com/paper/2407.00870/) (2024)
- [Cactus: Towards Psychological Counseling Conversations using Cognitive Behavioral Theory](https://mlsanity.com/paper/2407.03103/) (2024)
- [ALLaM: Large Language Models for Arabic and English](https://mlsanity.com/paper/2407.15390/) (2024)
- [Large Language Models Assume People are More Rational than We Really are](https://mlsanity.com/paper/2406.17055/) (2024)
- [GenSim: A General Social Simulation Platform with Large Language Model based Agents](https://mlsanity.com/paper/2410.04360/) (2024)
- [AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios](https://mlsanity.com/paper/2410.19346/) (2024)
- [Benchmarks as Microscopes: A Call for Model Metrology](https://mlsanity.com/paper/2407.16711/) (2024)
- [AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agents](https://mlsanity.com/paper/2409.09013/) (2024)
- [Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges](https://mlsanity.com/paper/2502.12378/) (2025)
- [Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agents](https://mlsanity.com/paper/2502.11355/) (2025)
- [InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Context](https://mlsanity.com/paper/2406.12203/) (2024)
- [Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?](https://mlsanity.com/paper/2410.23426/) (2024)
- [LLM-Generated Black-box Explanations Can Be Adversarially Helpful](https://mlsanity.com/paper/2405.06800/) (2024)
- [BattleAgent: Multi-modal Dynamic Emulation on Historical Battles to Complement Historical Analysis](https://mlsanity.com/paper/2404.15532/) (2024)
- [Do Large Language Models Align with Core Mental Health Counseling Competencies?](https://mlsanity.com/paper/2410.22446/) (2024)
- [Can A Society of Generative Agents Simulate Human Behavior and Inform Public Health Policy? A Case Study on Vaccine Hesitancy](https://mlsanity.com/paper/2503.09639/) (2025)
- [Autonomous Agents for Collaborative Task under Information Asymmetry](https://mlsanity.com/paper/2406.14928/) (2024)
- [Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue](https://mlsanity.com/paper/2409.08330/) (2024)
- [SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agents](https://mlsanity.com/paper/2502.15538/) (2025)
- [Cultural Learning-Based Culture Adaptation of Language Models](https://mlsanity.com/paper/2504.02953/) (2025)
- [Social World Models](https://mlsanity.com/paper/2509.00559/) (2025)
- [Social Skill Training with Large Language Models](https://mlsanity.com/paper/2404.04204/) (2024)
- [The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies](https://mlsanity.com/paper/2509.18052/) (2025)
- [Generative Value Conflicts Reveal LLM Priorities](https://mlsanity.com/paper/2509.25369/) (2025)
- [Cohesive Conversations: Enhancing Authenticity in Multi-Agent Simulated Dialogues](https://mlsanity.com/paper/2407.09897/) (2024)

## Abstract (arXiv)

Recent advances in large language models (LLM) have enabled richer social simulations, allowing for the study of various social phenomena. However, most recent work has used a more omniscient perspective on these simulations (e.g., single LLM to generate all interlocutors), which is fundamentally at odds with the non-omniscient, information asymmetric interactions that involve humans and AI agents in the real world. To examine these differences, we develop an evaluation framework to simulate social interactions with LLMs in various settings (omniscient, non-omniscient). Our experiments show that LLMs perform better in unrealistic, omniscient simulation settings but struggle in ones that more accurately reflect real-world conditions with information asymmetry. Our findings indicate that addressing information asymmetry remains a fundamental challenge for LLM-based agents.