# MirrorBench: A Benchmark to Evaluate Conversational User-Proxy Agents for Human-Likeness

arXiv: https://arxiv.org/abs/2601.08118 · HTML: https://mlsanity.com/paper/2601.08118/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 46 references that are in the map.
- First posted on arXiv 13 Jan 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 21 of 21)

- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024)
- [Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges](https://mlsanity.com/paper/2406.12624/) (2024)
- [Agentic Misalignment: How LLMs Could Be Insider Threats](https://mlsanity.com/paper/2510.05179/) (2025)
- [Asking Clarifying Questions in Open-Domain Information-Seeking Conversations](https://mlsanity.com/paper/1907.06554/) (2019)
- [Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions](https://mlsanity.com/paper/2109.05794/) (2021)
- [Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs](https://mlsanity.com/paper/2403.05020/) (2024)
- [SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?](https://mlsanity.com/paper/2510.05444/) (2025)
- [Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profiles](https://mlsanity.com/paper/2502.18968/) (2025)
- [SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluation](https://mlsanity.com/paper/2504.12722/) (2025)
- [User Simulation for Evaluating Information Access Systems](https://mlsanity.com/paper/2306.08550/) (2023)
- [Searching for Privacy Risks in LLM Agents via Simulation](https://mlsanity.com/paper/2508.10880/) (2025)
- [Very Large-Scale Multi-Agent Simulation in AgentScope](https://mlsanity.com/paper/2407.17789/) (2024)
- [Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs More Realistic and Less Risky](https://mlsanity.com/paper/2507.03336/) (2025)
- [Configurable multi-agent framework for scalable and realistic testing of llm-based agents](https://mlsanity.com/paper/2507.14705/) (2025)
- [DialogueForge: LLM Simulation of Human-Chatbot Dialogue](https://mlsanity.com/paper/2507.15752/) (2025)

## Built on by (most cited first, 7 of 7)

- [OdysSim: Building Foundation Models for Human Behavior Simulation](https://mlsanity.com/paper/2606.14199/) (2026)
- [Reinforcing Human Behavior Simulation via Verbal Feedback](https://mlsanity.com/paper/2605.20506/) (2026)
- [SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science](https://mlsanity.com/paper/2605.18630/) (2026)
- [Intent Speaks Louder: Controllable User Simulation Beyond Response Imitation](https://mlsanity.com/paper/2608.09420/) (2026)
- [Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation](https://mlsanity.com/paper/2607.27816/) (2026)
- [Investigating Assistant Bias in LLM User Simulators Using a Role Vector](https://mlsanity.com/paper/2609.00608/) (2026)
- [Back in Style: A Sociolinguistic Approach to Authoring and Measuring Persona Fidelity in User Simulation](https://mlsanity.com/paper/2610.10988/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used as human simulators, both for evaluating conversational systems and for generating fine-tuning data. However, naive "act-as-a-user" prompting often yields verbose, unrealistic utterances, motivating principled evaluation of *user proxy agents*. We present **MirrorBench**, a reproducible and extensible benchmarking framework that evaluates user proxies solely on their ability to produce human-like user utterances across diverse conversational regimes, explicitly decoupled from downstream task success. **MirrorBench** combines three lexical-diversity metrics (**MATTR**, **Yule's~K**, and **HD-D**) with three LLM-judge-based metrics (**GTEval**, **Pairwise Indistinguishability**, and **Rubric-and-Reason**), and contextualizes judge scores using Human-Human and Proxy-Proxy calibration controls. Across four public datasets, **MirrorBench** yields variance-aware comparisons and reveals systematic gaps between user proxies and real human users. The framework is open sourced at https://github.com/SAP/mirrorbench and includes a command-line interface for running and managing user-proxy benchmarking experiments.