mlsanityData to 14 Oct 2026

Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?

arXiv 2410.23426 ↗First posted 30 Oct 2024cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are increasingly employed for simulations, enabling applications in role-playing agents and Computational Social Science (CSS). However, the reliability of these simulations is under-explored, which raises concerns about the trustworthiness of LLMs in these applications. In this paper, we aim to answer ``How reliable is LLM-based simulation?'' To address this, we introduce TrustSim, an evaluation dataset covering 10 CSS-related topics, to systematically investigate the reliability of the LLM simulation. We conducted experiments on 14 LLMs and found that inconsistencies persist in the LLM-based simulated roles. In addition, the consistency level of LLMs does not strongly correlate with their general performance. To enhance the reliability of LLMs in simulation, we proposed Adaptive Learning Rate Based ORPO (AdaORPO), a reinforcement learning-based algorithm to improve the reliability in simulation across 7 LLMs. Our research provides a foundation for future studies to explore more robust and trustworthy LLM-based simulations.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 34 of its 61 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

34 of 61 references.

References · 34

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  4. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
  5. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  6. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 301
  7. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  8. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 241
  9. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 231
  10. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 218
  11. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 217
  12. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
22 more
  1. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  2. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  3. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 185
  4. 2023InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviewscited by 179
  5. 2023EconAgent: Large Language Model-Empowered Agents for Simulating Macroeconomic Activitiescited by 154
  6. 2023War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Warscited by 145
  7. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 130
  8. 2024Simulating Classroom Education with LLM-Empowered Agentscited by 127
  9. 2024How Well Can LLMs Negotiate? NegotiationArena Platform and Analysiscited by 100
  10. 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93
  11. 2023Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactionscited by 70
  12. 2024Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMscited by 59
  13. 2024Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluationcited by 58
  14. 2024Designing Heterogeneous LLM Agents for Financial Sentiment Analysiscited by 48
  15. 2024MathVC: An LLM-Simulated Multi-Character Virtual Classroom for Mathematics Educationcited by 37
  16. 2024LLM Harmony: Multi-Agent Communication for Problem Solvingcited by 35
  17. 2024How Reliable is Your Simulator? Analysis on the Limitations of Current LLM-based User Simulators for Conversational Recommendationcited by 27
  18. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 23
  19. 2024GermanPartiesQA: Benchmarking Commercial Large Language Models and AI Companions for Political Alignment and Sycophancycited by 14
  20. 2024Simulating The U.S. Senate: An LLM-Driven Agent Approach to Modeling Legislative Behavior and Bipartisanshipcited by 9
  21. 2024Revealing and Mitigating the Challenge of Detecting Character Knowledge Errors in LLM Role-Playingcited by 9
  22. 2024Can Poverty Be Reduced by Acting on Discrimination? An Agent-based Model for Policy Makingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 18; 2026: 5.

26 papers.

Sort:

By area:

  1. 2025AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Societycited by 186
  2. 2026This human study did not involve human subjects: Validating LLM simulations as behavioral evidencecited by 29
  3. 2024Large Language Models for Market Research: A Data-augmentation Approachcited by 20
  4. 2025Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Studycited by 17
  5. 2025Are Rules Meant to be Broken? Understanding Multilingual Moral Reasoning as a Computational Pipeline with UniMoralcited by 14
  6. 2025The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societiescited by 14
  7. 2025Simulating Society Requires Simulating Thoughtcited by 13
  8. 2024Piecing It All Together: Verifying Multi-Hop Multimodal Claimscited by 12
  9. 2025LLM-Based Social Simulations Require a Boundarycited by 12
  10. 2025The Hidden Strength of Disagreement: Unraveling the Consensus-Diversity Tradeoff in Adaptive Multi-Agent Systemscited by 11
  11. 2025SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environmentscited by 11
  12. 2025Artificial Intelligence in Spectroscopy: Advancing Chemistry from Prediction to Generation and Beyondcited by 6
14 more
  1. 2025Do Role-Playing Agents Practice What They Preach? Belief-Behavior Consistency in LLM-Based Simulations of Human Trustcited by 6
  2. 2025Generative Psycho-Lexical Approach for Constructing Value Systems in Large Language Modelscited by 4
  3. 2025Comparing Exploration-Exploitation Strategies of LLMs and Humans: Insights from Standard Multi-armed Bandit Experimentscited by 4
  4. 2025Investigating Tax Evasion Emergence Using Dual Large Language Model and Deep Reinforcement Learning Powered Agent-based Simulationcited by 3
  5. 2024Assessing the Impact of Conspiracy Theories Using Large Language Modelscited by 2
  6. 2026Classroom AI: Large Language Models as Grade-Specific Teacherscited by 2
  7. 2026Enhancing Persona Following at Decoding Time via Dynamic Importance Estimation for Role-Playing Agentscited by 2
  8. 2026PreScam: A Benchmark for Predicting Scam Progression from Early Conversationscited by 2
  9. 2025LLMs are Introvertcited by 1
  10. 2025Bridging Technology and Humanities: Evaluating the Impact of Large Language Models on Social Sciences Research with DeepSeek-R1cited by 0
  11. 2025Not There Yet: Evaluating Vision Language Models in Simulating the Visual Perception of People with Low Visioncited by 0
  12. 2025Knowing Your Uncertainty -- On the application of LLM in social sciencescited by 0
  13. 2025On the Role of Contextual Information and Ego States in LLM Agent Behavior for Transactional Analysis Dialoguescited by 0
  14. 2026RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficultycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Should LLM Agents Decide in Social Simulations? Comparing Finite-State and LLM-Based Decision Policies
  2. 2024LLM-Augmented Agent-Based Modelling for Social Simulations: Challenges and Opportunities
  3. 2025From the Fluency Fallacy to the Micro-to-Macro Validity Gap: Opportunities and Pitfalls of LLMs in Social Simulation
  4. 2024Systematic Biases in LLM Simulations of Debates
  5. 2024GenSim: A General Social Simulation Platform with Large Language Model based Agents
  6. 2025Evaluating the Use of Large Language Models as Synthetic Social Agents in Social Science Research
  7. 2024Sense and Sensitivity: Evaluating the simulation of social dynamics via Large Language Models
  8. 2026AI Agents Alone Are Not (Yet) Sufficient for Social Simulation