# SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?

arXiv: https://arxiv.org/abs/2510.05444 · HTML: https://mlsanity.com/paper/2510.05444/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2025.

## Built on by (most cited first, 30 of 50)

- [Mind the Sim2Real Gap in User Simulation for Agentic Tasks](https://mlsanity.com/paper/2603.11245/) (2026)
- [Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations](https://mlsanity.com/paper/2601.17087/) (2026)
- [HumanLM: Simulating Users with State Alignment Beats Response Imitation](https://mlsanity.com/paper/2603.03303/) (2026)
- [Simulated Students in Tutoring Dialogues: Substance or Illusion?](https://mlsanity.com/paper/2601.04025/) (2026)
- [OdysSim: Building Foundation Models for Human Behavior Simulation](https://mlsanity.com/paper/2606.14199/) (2026)
- [SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions](https://mlsanity.com/paper/2606.30573/) (2026)
- [Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants](https://mlsanity.com/paper/2605.09808/) (2026)
- [Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents](https://mlsanity.com/paper/2605.12894/) (2026)
- [LLMs Get Lost in Evolving User Intent](https://mlsanity.com/paper/2607.20734/) (2026)
- [MT-PingEval: Evaluating Multi-Turn Collaboration with Private Information Games](https://mlsanity.com/paper/2602.24188/) (2026)
- [MirrorBench: A Benchmark to Evaluate Conversational User-Proxy Agents for Human-Likeness](https://mlsanity.com/paper/2601.08118/) (2026)
- [MatrAIx: Simulating the World with 8.3 Billion Persona Agents](https://mlsanity.com/paper/2608.04205/) (2026)
- [DiscoverLLM: From Executing Intents to Discovering Them](https://mlsanity.com/paper/2602.03429/) (2026)
- [Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors](https://mlsanity.com/paper/2605.07847/) (2026)
- [The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Study](https://mlsanity.com/paper/2605.20767/) (2026)
- [Bridging the Agent-World Gap: Text World Models for LLM-based Agents](https://mlsanity.com/paper/2606.09032/) (2026)
- [Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction](https://mlsanity.com/paper/2603.23550/) (2026)
- [Reinforcing Human Behavior Simulation via Verbal Feedback](https://mlsanity.com/paper/2605.20506/) (2026)
- [GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents](https://mlsanity.com/paper/2609.12191/) (2026)
- [Knowing Isn't Understanding: Re-grounding Generative Proactivity with Epistemic and Behavioral Insight](https://mlsanity.com/paper/2602.15259/) (2026)
- [Auditing Support Strategies in LLMs through Grounded Multi-Turn Social Simulation](https://mlsanity.com/paper/2604.17079/) (2026)
- [SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science](https://mlsanity.com/paper/2605.18630/) (2026)
- [Investigating and Alleviating Harm Amplification in LLM Interactions](https://mlsanity.com/paper/2606.02423/) (2026)
- [VISTA: A Versatile Interactive User Simulation Toolkit for Agent Evaluation](https://mlsanity.com/paper/2606.11079/) (2026)
- [Controllable User Simulation](https://mlsanity.com/paper/2605.11519/) (2026)
- [Intent Speaks Louder: Controllable User Simulation Beyond Response Imitation](https://mlsanity.com/paper/2608.09420/) (2026)
- [KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn](https://mlsanity.com/paper/2608.17150/) (2026)
- [Disclosure-Gated User Simulation for Companion-Agent Evaluation](https://mlsanity.com/paper/2609.00982/) (2026)
- [UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training](https://mlsanity.com/paper/2609.38043/) (2026)
- [Beyond the Assistant Turn: User Turn Generation as a Probe of Interaction Awareness in Language Models](https://mlsanity.com/paper/2604.02315/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used in interactive applications, and human evaluation remains the gold standard for assessing their performance in multi-turn conversations. Since human studies are costly, time-consuming, and hard to reproduce, recent work explores using LLMs to simulate users for automatic assistant evaluation. However, there is no benchmark or systematic study to evaluate whether these simulated users are reliable stand-ins for real users. To address this, we introduce SimulatorArena, a benchmark of 909 annotated human-LLM conversations on two interactive tasks -- math tutoring and document creation. SimulatorArena evaluates simulators based on how closely their messages match human behavior and how well their assistant ratings align with human judgments. Experiments on various simulator methods show that simulators conditioned on user profiles, capturing traits like background and message styles, align closely with human judgments. They reach Spearman's ρ of 0.7 on both tasks, providing a practical, scalable alternative to human evaluation. Using the best simulator for each task, we benchmark 18 assistants, including the latest LLMs such as GPT-5, Claude 4.1 Opus, and Gemini 2.5 Pro.