# Stop Playing the Guessing Game! Target-free User Simulation for Evaluating Conversational Recommender Systems

arXiv: https://arxiv.org/abs/2411.16160 · HTML: https://mlsanity.com/paper/2411.16160/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2024.

## Built on by (most cited first, 4 of 4)

- [MT-RAIG: Novel Benchmark and Evaluation Framework for Retrieval-Augmented Insight Generation over Multiple Tables](https://mlsanity.com/paper/2502.11735/) (2025)
- [P-Check: Advancing Personalized Reward Model via Learning to Generate Dynamic Checklist](https://mlsanity.com/paper/2601.02986/) (2026)
- [LLM-Based User Simulation for Low-Knowledge Shilling Attacks on Recommender Systems](https://mlsanity.com/paper/2505.13528/) (2025)
- [RPM: Reasoning-Level Personalization for Black-Box Large Language Models](https://mlsanity.com/paper/2505.21082/) (2025)

## Abstract (arXiv)

Recent approaches in Conversational Recommender Systems (CRSs) have tried to simulate real-world users engaging in conversations with CRSs to create more realistic testing environments that reflect the complexity of human-agent dialogue. Despite the significant advancements, reliably evaluating the capability of CRSs to elicit user preferences still faces a significant challenge. Existing evaluation metrics often rely on target-biased user simulators that assume users have predefined preferences, leading to interactions that devolve into simplistic guessing game. These simulators typically guide the CRS toward specific target items based on fixed attributes, limiting the dynamic exploration of user preferences and struggling to capture the evolving nature of real-user interactions. Additionally, current evaluation metrics are predominantly focused on single-turn recall of target items, neglecting the intermediate processes of preference elicitation. To address this, we introduce PEPPER, a novel CRS evaluation protocol with target-free user simulators constructed from real-user interaction histories and reviews. PEPPER enables realistic user-CRS dialogues without falling into simplistic guessing games, allowing users to gradually discover their preferences through enriched interactions, thereby providing a more accurate and reliable assessment of the CRS's ability to elicit personal preferences. Furthermore, PEPPER presents detailed measures for comprehensively evaluating the preference elicitation capabilities of CRSs, encompassing both quantitative and qualitative measures that capture four distinct aspects of the preference elicitation process. Through extensive experiments, we demonstrate the validity of PEPPER as a simulation environment and conduct a thorough analysis of how effectively existing CRSs perform in preference elicitation and recommendation.