mlsanityData to 14 Oct 2026

Benchmarking Psychological Dynamics in Generative Agents

arXiv 2610.04246 ↗First posted 3 Oct 2026cs.HC cs.AIcited by 0

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

Large language models (LLMs) are increasingly deployed to simulate human behavior, acting as computational replicas of human subjects. Yet the lived psychological experience of humans is difficult to benchmark, particularly as it unfolds over time. We introduce a psychometric benchmark for computational replicas: personas that carry a fixed identity through an evolving sequence of events. Built entirely from published norms and meta-analytic effects, the benchmark scores two dimensions of psychological realism. The first, internal validity, quantifies whether generated trajectories reproduce the internal structure of repeated human measurement: distributions, the between- versus within-person variance partition, temporal dependence, and range. The second, external validity, quantifies whether replicas recover established trait, state, and indicator relations. Across 36 open-weight and proprietary LLMs from nine developers (1B-671B parameters), most recover the direction of established relations (84.3% mean agreement) and the variance partition (26 of 34), yet the within-person correlation and distributional structure elude recovery. Internal validity is independent of scale and capability: a mid-size open LLM (Gemma-3-27B) strikes the best trade-off between the two dimensions. The benchmark is a precondition for using computational replicas in causal inference across domains (e.g., marketing, healthcare), and identifies within-person grounding as the central challenge ahead.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 43 of its 89 references that are in the map. First posted on arXiv 3 Oct 2026.

Builds on

43 of 89 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026SPASM: Stable Persona-driven Agent Simulation for Multi-turn Dialogue Generation10 Apr
  2. 2025Are LLM Agents Behaviorally Coherent? Latent Profiles for Social Simulation3 Sep
  3. 2025Beyond Static Responses: Multi-Agent LLM Systems as a New Paradigm for Social Science Research2 Jun
  4. 2025AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society12 Feb
  5. 2023Generative Agents: Interactive Simulacra of Human Behavior7 Apr

Other references · 38

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  4. 2023Whose Opinions Do Language Models Reflect?cited by 773
  5. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  6. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
  7. 2022Using cognitive psychology to understand GPT-3cited by 398
  8. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  9. 2024Alignment faking in large language modelscited by 360
  10. 2023Role-Play with Large Language Modelscited by 294
  11. 2024Frontier Models are Capable of In-context Schemingcited by 287
  12. 2023How is ChatGPT's behavior changing over time?cited by 270
26 more
  1. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 234
  2. 2023Personality Traits in Large Language Modelscited by 169
  3. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
  4. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 144
  5. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
  6. 2025LLM Social Simulations Are a Promising Research Methodcited by 99
  7. 2023Questioning the Survey Responses of Large Language Modelscited by 74
  8. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 58
  9. 2024Large Language Models Assume People are More Rational than We Really arecited by 54
  10. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviorscited by 52
  11. 2025Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancementcited by 45
  12. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Modelscited by 39
  13. 2025Finetuning LLMs for Human Behavior Prediction in Social Science Experimentscited by 39
  14. 2025Consistently Simulating Human Personas with Multi-Turn Reinforcement Learningcited by 39
  15. 2026This human study did not involve human subjects: Validating LLM simulations as behavioral evidencecited by 29
  16. 2025Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarkscited by 28
  17. 2025Twin-2K-500: A dataset for building digital twins of over 2,000 people based on their answers to over 500 questionscited by 22
  18. 2025The threat of analytic flexibility in using large language models to simulate human datacited by 15
  19. 2024Do LLMs have Consistent Values?cited by 14
  20. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 13
  21. 2026OdysSim: Building Foundation Models for Human Behavior Simulationcited by 13
  22. 2026Persona Generators: Generating Diverse Synthetic Personas for Arbitrary Contextscited by 6
  23. 2026LLM Powered Social Digital Twins: A Framework for Simulating Population Behavioral Response to Policy Interventionscited by 4
  24. 2026The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safetycited by 2
  25. 2026Stable Personas: Dual-Assessment of Temporal Stability in LLM-Based Human Simulationcited by 2
  26. 2026Toward an AI-Powered Computational Testbed for Workforce Policycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Applying Psychometrics to Large Language Model Simulated Populations: Recreating the HEXACO Personality Inventory Experiment with Generative Agents
  2. 2026PTCBENCH: Benchmarking Contextual Stability of Personality Traits in LLM Systems
  3. 2026Mitigating Identity Essentialism in LLM Agents with Longitudinal Life Trajectories
  4. 2026MicroVerse: An Instrument for Measuring Self-Authored Identity Drift in Long-Horizon Multi-Agent Language-Model Simulations
  5. 2026Can Large Language Models Simulate Human Cognition Beyond Behavioral Imitation?
  6. 2025Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications
  7. 2025Mind the Gap: The Divergence Between Human and LLM-Generated Tasks
  8. 2026Do AI Personas Grow? Analyzing and Benchmarking Personality Evolution in LLM Agents After Life Events