mlsanityData to 14 Oct 2026

Validated Hypotheses as a Lens for Human-Likeness Evaluation in AI Agents

arXiv 2605.15473 ↗First posted 14 May 2026cs.CYcited by 0

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

We propose using validated behavioral hypotheses as a lens for evaluating human-likeness in LLM-based agents. Our key idea is simple: If an agent is human-like, a population of such agents should reach the same inferential conclusion as the human population when run through the same experiment. Decades of social science have produced many such validated findings, each anchored to concrete experimental protocols and robustly established through independent replication. This yields an evaluation that is objective, decomposable, and scalable. We operationalize this lens through HumanStudy-Bench, an open platform that turns published human-subject studies into reusable simulation environments and administers the evaluation to configurable agents. It scores agent-human alignment on two metrics: the Probability Alignment Score (PAS) for inferential agreement and the Effect Consistency Score (ECS) for effect-size agreement. We curated an initial suite of 12 studies whose hypotheses are robustly established through independent replication, and evaluated 10 models under 4 agent designs. Results show that agent responses polarize between full replication and complete failure; agent design influences alignment more than model scale, but its effect is non-monotonic.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 27 of its 88 references that are in the map. First posted on arXiv 14 May 2026.

Builds on

27 of 88 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025CoBRA: Programming Cognitive Bias in Social Agents Using Classic Social Science Experiments16 Sep
  2. 2025SocioVerse: A World Model for Social Simulation Powered by LLM Agents and A Pool of 10 Million Real-World Users14 Apr
  3. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents4 Dec
  4. 2024Exploring Prosocial Irrationality for LLM Agents: A Social Cognition View23 May
  5. 2023Character-LLM: A Trainable Agent for Role-Playing16 Oct

Other references · 22

  1. 2023Whose Opinions Do Language Models Reflect?cited by 773
  2. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  3. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
  4. 2022Using cognitive psychology to understand GPT-3cited by 398
  5. 2023Role-Play with Large Language Modelscited by 294
  6. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  7. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 234
  8. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
  9. 2024Quantifying the Persona Effect in LLM Simulationscited by 169
  10. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 144
  11. 2022Evaluating and Inducing Personality in Pre-trained Language Modelscited by 123
  12. 2022Who is GPT-3? An Exploration of Personality, Values and Demographicscited by 112
10 more
  1. 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 66
  2. 2023The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspectivecited by 59
  3. 2025Specializing Large Language Models to Simulate Survey Response Distributions for Global Populationscited by 53
  4. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviorscited by 52
  5. 2024Virtual Personas for Language Models via an Anthology of Backstoriescited by 51
  6. 2025The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Modelscited by 42
  7. 2025Finetuning LLMs for Human Behavior Prediction in Social Science Experimentscited by 39
  8. 2025Do Language Models Mirror Human Confidence? Exploring Psychological Insights to Address Overconfidence in LLMscited by 19
  9. 2024Can AI with High Reasoning Ability Replicate Human-like Decision Making in Economic Experiments?cited by 9
  10. 2024Simulating Field Experiments with Large Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026HumanStudy-Bench: Towards AI Agent Design for Participant Simulation
  2. 2026SPEAR: Five Principles for Interactive Human-Agent Alignment
  3. 2026TruthTensor: Evaluating LLMs through Human Imitation on Prediction Market under Drift and Holistic Reasoning
  4. 2025HugAgent: A Human Simulation Benchmark for Individual-Level Reasoning
  5. 2026CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulation
  6. 2025The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation
  7. 2026Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment
  8. 2025VeriLA: A Human-Centered Evaluation Framework for Interpretable Verification of LLM Agent Failures