# UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems

arXiv: https://arxiv.org/abs/2301.05544 · HTML: https://mlsanity.com/paper/2301.05544/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jan 2023.

## Built on by (most cited first, 11 of 11)

- [Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions](https://mlsanity.com/paper/2312.02913/) (2023)
- [Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models](https://mlsanity.com/paper/2305.13112/) (2023)
- [Interactive Agents: Simulating Counselor-Client Psychological Counseling via Role-Playing LLM-to-LLM Interactions](https://mlsanity.com/paper/2408.15787/) (2024)
- [User Simulation for Evaluating Information Access Systems](https://mlsanity.com/paper/2306.08550/) (2023)
- [ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders](https://mlsanity.com/paper/2602.16938/) (2026)
- [Concept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factors](https://mlsanity.com/paper/2404.03304/) (2024)
- [Human vs. Agent in Task-Oriented Conversations](https://mlsanity.com/paper/2509.17619/) (2025)
- [CXSimulator: A User Behavior Simulation using LLM Embeddings for Web-Marketing Campaign Assessment](https://mlsanity.com/paper/2407.21553/) (2024)
- [Multi-trait User Simulation with Adaptive Decoding for Conversational Task Assistants](https://mlsanity.com/paper/2410.12891/) (2024)
- [Grading Conversational Responses Of Chatbots](https://mlsanity.com/paper/2303.12038/) (2023)
- [Synthetic Dialogue Generation for Interactive Conversational Elicitation & Recommendation (ICER)](https://mlsanity.com/paper/2510.02331/) (2025)

## Abstract (arXiv)

We present an extensible user simulation toolkit to facilitate automatic evaluation of conversational recommender systems. It builds on an established agenda-based approach and extends it with several novel elements, including user satisfaction prediction, persona and context modeling, and conditional natural language generation. We showcase the toolkit with a pre-existing movie recommender system and demonstrate its ability to simulate dialogues that mimic real conversations, while requiring only a handful of manually annotated dialogues as training data.