# Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data

arXiv: https://arxiv.org/abs/2503.20749 · HTML: https://mlsanity.com/paper/2503.20749/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 31 references that are in the map.
- First posted on arXiv 26 Mar 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 12 of 12)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis](https://mlsanity.com/paper/2307.12856/) (2023)
- [CollabLLM: From Passive Responders to Active Collaborators](https://mlsanity.com/paper/2502.00640/) (2025)
- [RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation](https://mlsanity.com/paper/2402.16667/) (2024)
- [UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design](https://mlsanity.com/paper/2502.12561/) (2025)
- [WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents](https://mlsanity.com/paper/2404.05902/) (2024)
- [Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents](https://mlsanity.com/paper/2502.13012/) (2025)
- [ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agents](https://mlsanity.com/paper/2411.00927/) (2024)

## Built on by (most cited first, 28 of 28)

- [Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations](https://mlsanity.com/paper/2601.17087/) (2026)
- [HumanLM: Simulating Users with State Alignment Beats Response Imitation](https://mlsanity.com/paper/2603.03303/) (2026)
- [AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agents](https://mlsanity.com/paper/2504.09723/) (2025)
- [Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning](https://mlsanity.com/paper/2507.17842/) (2025)
- [LLM Agent Meets Agentic AI: Can LLM Agents Simulate Customers to Evaluate Agentic-AI-based Shopping Assistants?](https://mlsanity.com/paper/2509.21501/) (2025)
- [UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agents](https://mlsanity.com/paper/2504.09407/) (2025)
- [Learning User Simulators with Turing Rewards](https://mlsanity.com/paper/2606.19336/) (2026)
- [ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents](https://mlsanity.com/paper/2605.16116/) (2026)
- [Simulating Human Memory with Language Models](https://mlsanity.com/paper/2605.25680/) (2026)
- [LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs](https://mlsanity.com/paper/2608.05246/) (2026)
- [LaajMeter: A Framework for LaaJ Evaluation](https://mlsanity.com/paper/2508.10161/) (2025)
- [SimPersona: Learning Discrete Buyer Personas from Raw Clickstreams for Grounded E-Commerce Agents](https://mlsanity.com/paper/2605.14205/) (2026)
- [Iterating Toward Better Search: A Two-Agent Simulation Framework for Evaluating Agentic Search Architectures in E-Commerce](https://mlsanity.com/paper/2606.12924/) (2026)
- [Large Behavior Model: A Promptable Digital Twin of the Retail Customer](https://mlsanity.com/paper/2607.06993/) (2026)
- [When Can LLM Digital Twins Reduce Human Measurement? From Behavioral Fidelity to Statistical Substitutability](https://mlsanity.com/paper/2609.07987/) (2026)
- [SOCIA-Nabla: Textual Gradient Meets Multi-Agent Orchestration for Automated Simulator Generation](https://mlsanity.com/paper/2510.18551/) (2025)
- [SafeTalkCoach: Diversity-Driven Multi-Agent Simulation for Parent-Teen Health Conversations](https://mlsanity.com/paper/2602.00017/) (2026)
- [From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs](https://mlsanity.com/paper/2604.14137/) (2026)
- [Recon: Reconstruction-Guided Reasoning Synthesis for User Modeling](https://mlsanity.com/paper/2605.26969/) (2026)
- [Humans Are More Diverse: Frontier LLMs Show Extreme Policies in Idealised AI Development Races](https://mlsanity.com/paper/2608.01193/) (2026)
- [Rethinking the Evaluation and Optimization of LLM-Based Social Simulation](https://mlsanity.com/paper/2608.19689/) (2026)
- [Automatic multimodal UX improvement recommendations from LLM agent user simulations](https://mlsanity.com/paper/2609.22971/) (2026)
- [From Offline Proxies to Online Decisions: A Layered Engagement Evaluation Framework for Conversational AI](https://mlsanity.com/paper/2609.25408/) (2026)
- [TRACER: Trajectory-Aligned Learning for Multi-Turn User Simulation](https://mlsanity.com/paper/2609.28690/) (2026)
- [Where Do Test-Time Scaling and Training Fall Short in Individual Stance Prediction?](https://mlsanity.com/paper/2609.33155/) (2026)
- [AnthroDial: Benchmarking LLM Anthropomorphism in Autonomous Social Interaction](https://mlsanity.com/paper/2609.37853/) (2026)
- [SimTrace: Grounded Multimodal User Trajectories Generation for Online User Modeling](https://mlsanity.com/paper/2609.38397/) (2026)
- [Training User Simulators with Latent Intrinsic Feedback](https://mlsanity.com/paper/2610.13617/) (2026)

## Abstract (arXiv)

Recent research shows that LLM Agents can generate ``believable'' human behaviors via prompt-only methods, and such agents have been increasingly adopted in downstream applications. However, existing evaluation of these agents only focuses on qualitative believability (whether human raters think they are accurate), leaving open questions of whether LLM agents can accurately generate step-by-step actions mimicking a particular human's behavior in a multi-turn interaction task. In this work, we take shopping as a case study and present the first large-scale quantitative evaluation of state-of-the-art LLMs' ability to accurately simulate human behavior. Using real-world data from 31,865 online shopping sessions containing 230,965 user actions, our evaluation reveals that prompt-based LLMs (DeepSeek-R1, Llama, Claude) achieve only 11.86% accuracy in generating human actions, highlighting a substantial gap in actual behavioral accuracy. Through experiments, we also showcase that strategies as simple as fine-tuning LLMs on real human click-through data augmented with synthesized reasoning traces can greatly enhance models' performance. The fine-tuned Qwen2.5-7B achieves 17.26% action generation accuracy and 33.86% F1 score on final purchase prediction, representing substantial improvements of 5.4% and 13.85% over prompt-only baselines. This work establishes the first rigorous benchmark for human behavior simulation and provides actionable insights for developing more accurate LLM agents for future downstream applications.