mlsanityData to 7 Oct 2026

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

arXiv 2603.11245 ↗First posted 11 Mar 2026cs.AIcited by 44

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

As NLP evaluation shifts from static benchmarks to multi-turn interactive settings, LLM-based simulators have become widely used as user proxies, serving two roles: generating user turns and providing evaluation signals. Yet, these simulations are frequently assumed to be faithful to real human behaviors, often without rigorous verification. We formalize the Sim2Real gap in user simulation and present the first study running the full τ-bench protocol with real humans (451 participants, 165 tasks), benchmarking 31 LLM simulators across proprietary, open-source, and specialized families using the User-Sim Index (USI), a metric we introduce to quantify how well LLM simulators resemble real user interactive behaviors and feedback. Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity, creating an "easy mode" that inflates agent success rates above the human baseline. In evaluations, real humans provide nuanced judgments across eight quality dimensions while simulated users produce uniformly more positive feedback; rule-based rewards are failing to capture rich feedback signals generated by human users. Overall, higher general model capability does not necessarily yield more faithful user simulation. These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle and motivate improved models for user simulation.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 37 of its 51 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

37 of 51 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations23 Jan
  2. 2024Agent-Based Modelling Meets Generative AI in Social Network Simulations25 Nov
  3. 2024PersonaGym: Evaluating Persona Agents and LLMs25 Jul
  4. 2023User Behavior Simulation with Large Language Model based Agents5 Jun

Other references · 33

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  4. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  5. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  6. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  7. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
  8. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  9. 2013Crowdsourcing a Word-Emotion Association Lexiconcited by 193
  10. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  11. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 145
  12. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
21 more
  1. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 69
  2. 2016A User Simulator for Task-Completion Dialoguescited by 56
  3. 2025Flipping the Dialogue: Training and Evaluating User Language Modelscited by 50
  4. 2021Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systemscited by 47
  5. 2024Evaluating Large Language Models as Generative User Simulators for Conversational Recommendationcited by 45
  6. 2025SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?cited by 45
  7. 2024HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactionscited by 42
  8. 2025Training Proactive and Personalized LLM Agentscited by 42
  9. 2025SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluationcited by 32
  10. 2024USimAgent: Large Language Models for Simulating Search Userscited by 31
  11. 2025Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineeringcited by 28
  12. 2025Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflowscited by 27
  13. 2025CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulationcited by 26
  14. 2024Reliable LLM-based User Simulator for Task-Oriented Dialogue Systemscited by 24
  15. 2024BASES: Large-scale Web Search User Simulation with Large Language Model based Agentscited by 24
  16. 2023User Simulation with Large Language Models for Evaluating Task-Oriented Dialoguecited by 19
  17. 2024Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialoguecited by 15
  18. 2022Detecting Text Formality: A Study of Text Classification Approachescited by 11
  19. 2025TOM-SWE: User Mental Modeling For Software Engineering Agentscited by 11
  20. 2025Human vs. Agent in Task-Oriented Conversationscited by 9
  21. 2025Exploring Human-Like Thinking in Search Simulations with Large Language Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 44.

44 papers.

Sort:

By area:

  1. 2026SWE-chat: Coding Agent Interactions From Real Users in the Wildcited by 36
  2. 2026RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulationcited by 8
  3. 2026OdysSim: Building Foundation Models for Human Behavior Simulationcited by 8
  4. 2026FutureSim: Replaying World Events to Evaluate Adaptive Agentscited by 7
  5. 2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 6
  6. 2026Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agentscited by 6
  7. 2026MatrAIx: Simulating the World with 8.3 Billion Persona Agentscited by 5
  8. 2026Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviorscited by 4
  9. 2026ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactionscited by 4
  10. 2026Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversationscited by 3
  11. 2026Bridging the Agent-World Gap: Text World Models for LLM-based Agentscited by 3
  12. 2026Learning User Simulators with Turing Rewardscited by 3
32 more
  1. 2026GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agentscited by 3
  2. 2026Simulating Human Memory with Language Modelscited by 2
  3. 2026DRACULA: Hunting for the Actions Users Want Deep Research Agents to Executecited by 1
  4. 2026An Empirical Study of Proactive Coding Assistants in Real-World Software Developmentcited by 1
  5. 2026CollabSkill: Evaluating Human-Agent Collaboration On Real-World Taskscited by 1
  6. 2026VISTA: A Versatile Interactive User Simulation Toolkit for Agent Evaluationcited by 1
  7. 2026ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectoriescited by 1
  8. 2026Learning to Prompt: Improving Student Engagement with Adaptive LLM-based High-School Tutoringcited by 1
  9. 2026Simulated Customers Never Walk Away: Decision Fidelity of LLM User Simulators Measured Against Real Purchase Outcomescited by 1
  10. 2026Large Behavior Model: A Promptable Digital Twin of the Retail Customercited by 1
  11. 2026Intent Speaks Louder: Controllable User Simulation Beyond Response Imitationcited by 1
  12. 2026One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RLcited by 1
  13. 2026Beyond the Assistant Turn: User Turn Generation as a Probe of Interaction Awareness in Language Modelscited by 0
  14. 2026CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulationcited by 0
  15. 2026BehaviorBench: Modeling Real-World User Decisions from Behavioral Tracescited by 0
  16. 2026Multi-Modal Agents for Power Distribution Defect Detection: An Evaluation of Foundation Modelscited by 0
  17. 2026Will Scaling Improve Social Simulation with LLMs?cited by 0
  18. 2026Where Knowledge and Authority Sit Changes What an Agent Benchmark Can Resolvecited by 0
  19. 2026MemoryForge: Synthesize Lifelong Memory for Human-Like LLM Agentscited by 0
  20. 2026Readable, Faithful, Used: Three Dissociable Properties of Demographic Identity in a Language Modelcited by 0
  21. 2026"Act Like a 5th Grader" is Not Enough: Bounding Knowledge in LLM-Based User Simulatorscited by 0
  22. 2026Investigating Assistant Bias in LLM User Simulators Using a Role Vectorcited by 0
  23. 2026Disclosure-Gated User Simulation for Companion-Agent Evaluationcited by 0
  24. 2026Measuring the Behavioral Fidelity of Long-Horizon Human Activity Simulationscited by 0
  25. 2026Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failurecited by 0
  26. 2026Evaluating Open-Weight E-Commerce Agents with Environment-Grounded Verificationcited by 0
  27. 2026Pretrained Persona Mixture Models and Tandem Models for Human Simulationcited by 0
  28. 2026PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learningcited by 0
  29. 2026Selective Elicitation as a Commercial Influence Channel: A Reproducible Synthetic Shopping-Agent Stress Testcited by 0
  30. 2026MIMESIS: Learning User Simulators as Training Environments for Interactive Agentscited by 0
  31. 2026Back in Style: A Sociolinguistic Approach to Authoring and Measuring Persona Fidelity in User Simulationcited by 0
  32. 2026InterviewPlayground: A Simulation Environment for Evaluating AI Interviewerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training
  2. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors
  3. 2025Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data
  4. 2024Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs
  5. 2025From the Fluency Fallacy to the Micro-to-Macro Validity Gap: Opportunities and Pitfalls of LLMs in Social Simulation
  6. 2025Don't Trust Generative Agents to Mimic Communication on Social Networks Unless You Benchmarked their Empirical Realism
  7. 2025Are LLM Agents Behaviorally Coherent? Latent Profiles for Social Simulation
  8. 2026Can Large Language Models Simulate Human Cognition Beyond Behavioral Imitation?