# Identifying Breakdowns in Conversational Recommender Systems using User Simulation

arXiv: https://arxiv.org/abs/2405.14249 · HTML: https://mlsanity.com/paper/2405.14249/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2024.

## Built on by (most cited first, 5 of 5)

- [Interactive Agents: Simulating Counselor-Client Psychological Counseling via Role-Playing LLM-to-LLM Interactions](https://mlsanity.com/paper/2408.15787/) (2024)
- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [User Simulation in the Era of Generative AI: User Modeling, Synthetic Data Generation, and System Evaluation](https://mlsanity.com/paper/2501.04410/) (2025)
- [RefuteBench 2.0 -- Agentic Benchmark for Dynamic Evaluation of LLM Responses to Refutation Instruction](https://mlsanity.com/paper/2502.18308/) (2025)
- [SimInter: Evidence-Grounded User Simulation with Interview-Based Profile Enrichment for Conversational Recommender Systems](https://mlsanity.com/paper/2610.18308/) (2026)

## Abstract (arXiv)

We present a methodology to systematically test conversational recommender systems with regards to conversational breakdowns. It involves examining conversations generated between the system and simulated users for a set of pre-defined breakdown types, extracting responsible conversational paths, and characterizing them in terms of the underlying dialogue intents. User simulation offers the advantages of simplicity, cost-effectiveness, and time efficiency for obtaining conversations where potential breakdowns can be identified. The proposed methodology can be used as diagnostic tool as well as a development tool to improve conversational recommendation systems. We apply our methodology in a case study with an existing conversational recommender system and user simulator, demonstrating that with just a few iterations, we can make the system more robust to conversational breakdowns.