mlsanityData to 7 Oct 2026

LLM-based Human Simulations Have Not Yet Been Reliable

arXiv 2501.08579 ↗First posted 15 Jan 2025cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are increasingly employed for simulating human behaviors across diverse domains. However, our position is that current LLM-based human simulations remain insufficiently reliable, as evidenced by significant discrepancies between their outcomes and authentic human actions. Our investigation begins with a systematic review of LLM-based human simulations in social, economic, policy, and psychological contexts, identifying their common frameworks, recent advances, and persistent limitations. This review reveals that such discrepancies primarily stem from inherent limitations of LLMs and flaws in simulation design, both of which are examined in detail. Building on these insights, we propose a systematic solution framework that emphasizes enriching data foundations, advancing LLM capabilities, and ensuring robust simulation design to enhance reliability. Finally, we introduce a structured algorithm that operationalizes the proposed framework, aiming to guide credible and human-aligned LLM-based simulations. To facilitate further research, we provide a curated list of related literature and resources at https://github.com/Persdre/awesome-llm-human-simulation.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 30 of its 83 references that are in the map. First posted on arXiv 15 Jan 2025.

Builds on

30 of 83 references.

References · 30

  1. 2024A Survey on LLM-as-a-Judgecited by 1,384
  2. 2022Using cognitive psychology to understand GPT-3cited by 389
  3. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  4. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  5. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 210
  6. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 196
  7. 2024Are We Done with MMLU?cited by 178
  8. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  9. 2024Quantifying the Persona Effect in LLM Simulationscited by 164
  10. 2023EconAgent: Large Language Model-Empowered Agents for Simulating Macroeconomic Activitiescited by 154
  11. 2024Hello Again! LLM-powered Personalized Agent for Long-term Dialoguecited by 121
  12. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 120
18 more
  1. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  2. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluationcited by 86
  3. 2023The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspectivecited by 57
  4. 2024Learning to Use Tools via Cooperative and Interactive Agentscited by 54
  5. 2023Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaborationcited by 48
  6. 2024LLM economicus? Mapping the Behavioral Biases of LLMs via Utility Theorycited by 44
  7. 2023Urban Generative Intelligence (UGI): A Foundational Platform for Agents in Embodied City Environmentcited by 37
  8. 2024AgentCourt: Simulating Court with Adversarial Evolvable Lawyer Agentscited by 36
  9. 2024Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlookcited by 33
  10. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  11. 2024PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agentscited by 28
  12. 2024How Reliable is Your Simulator? Analysis on the Limitations of Current LLM-based User Simulators for Conversational Recommendationcited by 25
  13. 2024Evaluating the Performance of Large Language Models via Debatescited by 19
  14. 2024Cognitive Overload Attack:Prompt Injection for Long Contextcited by 13
  15. 2024Improve Temporal Awareness of LLMs for Sequential Recommendationcited by 8
  16. 2024From Prejudice to Parity: A New Approach to Debiasing Large Language Model Word Embeddingscited by 3
  17. 2024The Economic Implications of Large Language Model Selection on Earnings and Return on Investment: A Decision Theoretic Modelcited by 2
  18. 2024She Works, He Works: A Curious Exploration of Gender Bias in AI-Generated Imagerycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 13.

30 papers.

Sort:

By area:

  1. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  2. 2025MasRouter: Learning to Route LLMs for Multi-Agent Systemscited by 80
  3. 2025Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agentscited by 22
  4. 2025Large Language Models Do Not Simulate Human Psychologycited by 17
  5. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
  6. 2025LLM-Based Social Simulations Require a Boundarycited by 11
  7. 2025Approximating Human Strategic Reasoning with LLM-Enhanced Recursive Reasoners Leveraging Multi-agent Hypergamescited by 7
  8. 2026Diversity Collapse in Multi-Agent LLM Systems: Structural Coupling and Collective Failure in Open-Ended Idea Generationcited by 7
  9. 2025Do Role-Playing Agents Practice What They Preach? Belief-Behavior Consistency in LLM-Based Simulations of Human Trustcited by 6
  10. 2025Beyond Brainstorming: What Drives High-Quality Scientific Ideas? Lessons from Multi-Agent Collaborationcited by 6
  11. 2025DPRF: A Generalizable Dynamic Persona Refinement Framework for Optimizing Behavior Alignment Between Personalized LLM Role-Playing Agents and Humanscited by 5
  12. 2025PulseReddit: A Novel Reddit Dataset for Benchmarking MAS in High-Frequency Cryptocurrency Tradingcited by 2
18 more
  1. 2025SimuPanel: A Novel Immersive Multi-Agent System to Simulate Interactive Expert Panel Discussioncited by 2
  2. 2026Cognitive Models and AI Algorithms Provide Templates for Designing Language Agentscited by 2
  3. 2026Simulating Human Memory with Language Modelscited by 2
  4. 2025Arti-"fickle" Intelligence: Using LLMs as a Tool for Inference in the Political and Social Sciencescited by 1
  5. 2025Expanding the Action Space of LLMs to Reason Beyond Languagecited by 1
  6. 2026HAG: Hierarchical Demographic Tree-based Agent Generation for Topic-Adaptive Simulationcited by 1
  7. 2026MARO: Learning Stronger Reasoning from Social Interactioncited by 1
  8. 2026Communication Enhances LLMs' Stability in Strategic Thinkingcited by 1
  9. 2026Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBenchcited by 1
  10. 2026Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agentscited by 1
  11. 2026Beyond Averages: Evaluating LLMs on Human Survey Replication at the Distributional Levelcited by 1
  12. 2025Artificial Utopia: Simulation and Intelligent Agents for a Democratised Futurecited by 0
  13. 2025Doppelganger Method: Breaking Role Consistency in LLM Agent via Prompt-based Transferable Adversarial Attackcited by 0
  14. 2025HiCoTraj:Zero-Shot Demographic Reasoning via Hierarchical Chain-of-Thought Prompting from Trajectorycited by 0
  15. 2026CAMO: An Agentic Framework for Automated Causal Discovery from Micro Behaviors to Macro Emergence in LLM Agent Simulationscited by 0
  16. 2026The Granularity Axis: A Micro-to-Macro Latent Direction for Social Roles in Language Modelscited by 0
  17. 2026Evaluating Language Models in Realistic Conversational Contextscited by 0
  18. 2026Cheap Talk Stabilizes Strategic Interaction in LLM Agentscited by 0