# UXBench Pro: Benchmarking Personalized User Experience in Multi-Turn Dialogue Interactions

arXiv: https://arxiv.org/abs/2610.11638 · HTML: https://mlsanity.com/paper/2610.11638/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 32 references that are in the map.
- First posted on arXiv 8 Oct 2026.

## Builds on (most cited first, 6 of 6)

- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [In-Context Learning User Simulators for Task-Oriented Dialog Systems](https://mlsanity.com/paper/2306.00774/) (2023)
- [OneMillion-Bench: How Far are Language Agents from Human Experts?](https://mlsanity.com/paper/2603.07980/) (2026)
- [Understanding Communication Preferences of Information Workers in Engagement with Text-Based Conversational Agents](https://mlsanity.com/paper/2410.20468/) (2024)
- [UXBench: Benchmarking User Experience in AI Assistants](https://mlsanity.com/paper/2606.09570/) (2026)
- [CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production](https://mlsanity.com/paper/2603.01973/) (2026)

## Abstract (arXiv)

Evaluating user experience (UX) with automated computational methods has gained increasing attention, supported by empirical evidence from UXBench. However, binary preference prediction provides limited insight, while relying on a single user-agnostic reward model overlooks the inherent heterogeneity of users, whose expectations can differ substantially. In this paper, we present UXBench Pro, comprising 1{,}000 test instances derived from real user interactions across 12 task scenarios and 82 domains. Each instance is paired with a FACTORS user profile that characterizes the user through seven interpretable behavioral facets, differentiating user groups. To provide richer evaluation insights, we introduce a dual-perspective paradigm that combines a personalized User Reward Model (URM) for third-person judgment with Sim4Eval, a user simulator that enables multi-turn interactions and provides first-person evaluation across four cognitive state dimensions. To assess the reliability of these based evaluators, we further introduce two meta-benchmarks, URMBench and USimBench, that evaluate how faithfully they reproduce real human preferences and behaviors. Extensive experiments reveal seven key findings that highlight the importance of user modeling and multi-perspective evaluation, offering a fresh perspective on user-centric benchmarking and motivating personalized model optimization.