AgentPersonaBench: Benchmarking Persona-Driven User Simulation
Not on the map. Why?
Abstract (arXiv)
We introduce AgentPersonaBench (APB), a benchmark evaluating whether persona conditioning faithfully steers downstream agent behavior. While language models are increasingly deployed for persona-driven user simulation, existing benchmarks primarily evaluate conversational styling or self-reports rather than authentic behavioral fidelity. APB evaluates latent persona adherence one trait at a time, embedding each target trait within a complete synthetic profile without explicitly naming the trait or disclosing the test. Ground-truth adherence is verified strictly from observable actions across four interaction surfaces of increasing realism: survey, chat, web (interactive web environments), and app (desktop software environments). APB comprises 2,460 tasks spanning 867 traits, verified through automated audits and expert review. Our evaluation of 20 frontier model arms demonstrates that high-fidelity user simulation is already attainable: leading models achieve up to 84.7% full-pass adherence under unprompted conditions. At the same time, APB identifies clear behavioral boundaries: adherence drops across interaction modalities (only 37.9-64.3% pass all four surfaces), multi-attribute demands degrade retention, and competing model families exhibit pronounced behavioral divergence.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 76 of its 90 references that are in the map. First posted on arXiv 3 Oct 2026.
Builds on
76 of 90 references.
References · 76
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
- 2020Learning to summarize from human feedbackcited by 2,588
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
38 more
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
- 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
- 2023GAIA: a benchmark for General AI Assistantscited by 1,128
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
- 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
- 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
- 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
- 2023Whose Opinions Do Language Models Reflect?cited by 773
- 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
- 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
- 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
- 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
- 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
- 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
- 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
- 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
- 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
- 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
- 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
- 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
- 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
- 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
- 2023Character-LLM: A Trainable Agent for Role-Playingcited by 339
- 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
- 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 234
- 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 223
- 2024AI Agents That Mattercited by 219
50 most cited of 76 All 76 (JSON)
Built on by
Not cited yet.