Simulated Customers Never Walk Away: Decision Fidelity of LLM User Simulators Measured Against Real Purchase Outcomes
On the map: LLM agents and tool use → Social simulation and multi-agent debate
Abstract (arXiv)
LLM-as-user-simulation has become core infrastructure for conversational AI: agent benchmarks (tau-bench), training pipelines, and a growing body of fidelity studies all rely on LLMs role-playing the human side of dialogue. Existing frameworks measure communicative fidelity -- whether simulators talk like humans -- against ground truth from paid participants role-playing assigned goals. We argue this has a structural blind spot: when the goal is assigned, the user's willingness is exogenous, so no framework can test whether simulators make decisions like real users whose motivation is endogenous, latent, and decaying. We introduce decision fidelity -- whether a simulated population reproduces the decision-state dynamics of real users facing real, consequential choices -- and measure it on a unique testbed: 2,790 production conversations between an LLM sales agent and real customers, including 793 with verified payment outcomes. Using a teacher-forced probe protocol that holds context and instrument fixed, we find a systematic, outcome-correlated failure we call the disengagement deficit: simulators reproduce eventual buyers almost exactly (depth bias +0.09) but inflate eventual non-buyers toward the purchase frame (depth bias +0.40; d=0.38, p<0.001), halving expressed resistance (25.1% to 13.5%) and nearly doubling deliberation (21.9% to 40.1%) while fabricating no purchases. The deficit replicates across model families (DeepSeek: d=0.41, p=0.002) and resists the obvious fix: instructing the simulator that it may disengage cuts marginal bias five-fold but barely moves the outcome-conditioned contrast (d=0.34, p=0.008). Real non-buyers say "not now" and stop; simulated non-buyers ask about price. Evaluating or training sales and persuasion agents against such simulators overstates funnel progress exactly where it matters most -- the customers who walk away.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 20 of its 23 references that are in the map. First posted on arXiv 16 Jun 2026.
Builds on
20 of 23 references.
Closest ancestors · 3
Same direction, newest first
- 2026RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation7 Apr
- 2026Mind the Sim2Real Gap in User Simulation for Agentic Tasks11 Mar
- 2026Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations23 Jan
Other references · 17
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2024A Survey on LLM-as-a-Judgecited by 1,384
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2019Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Goodcited by 207
- 2016A User Simulator for Task-Completion Dialoguescited by 56
- 2023Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Modelscited by 37
- 2026This human study did not involve human subjects: Validating LLM simulations as behavioral evidencecited by 26
- 2021RESPER: Computationally Modelling Resisting Strategies in Persuasive Conversationscited by 18
- 2026ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenderscited by 10
- 2026Evaluating Alignment of Behavioral Dispositions in LLMscited by 6
- 2025SalesRLAgent: A Reinforcement Learning Approach for Real-Time Sales Conversion Prediction and Optimizationcited by 2
5 more
- 2025Learning When to Quit in Sales Conversationscited by 2
- 2025AI-Salesman: Towards Reliable Large Language Model Driven Telemarketingcited by 2
- 2026The Autocorrelation Blind Spot: Why 42% of Turn-Level Findings in LLM Conversation Analysis May Be Spuriouscited by 2
- 2024User Willingness-aware Sales Talk Datasetcited by 1
- 2026How Much Does Persuasion Strategy Matter? LLM-Annotated Evidence from Charitable Donation Dialoguescited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Can Generative AI agents behave like humans? Evidence from laboratory market experiments
- 2026CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulation
- 2026Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents
- 2025Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Making
- 2026Decision-aware User Simulation Agent for Evaluating Conversational Recommender Systems
- 2026LLM Consumer Behavior Theory: Foundations of a Novel Research Field
- 2026TRACER: Trajectory-Aligned Learning for Multi-Turn User Simulation
- 2026GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents