# Evaluating Conversational Recommender Systems: A Landscape of Research

arXiv: https://arxiv.org/abs/2208.12061 · HTML: https://mlsanity.com/paper/2208.12061/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Aug 2022.

## Built on by (most cited first, 7 of 7)

- [Approximating Online Human Evaluation of Social Chatbots with Prompting](https://mlsanity.com/paper/2304.05253/) (2023)
- [Behavior Alignment: A New Perspective of Evaluating LLM-based Conversational Recommender Systems](https://mlsanity.com/paper/2404.11773/) (2024)
- [Talk the Walk: Synthetic Data Generation for Conversational Music Recommendation](https://mlsanity.com/paper/2301.11489/) (2023)
- [Evaluating User Experience in Conversational Recommender Systems: A Systematic Review Across Classical and LLM-Powered Approaches](https://mlsanity.com/paper/2508.02096/) (2025)
- [INFACT: An Online Human Evaluation Framework for Conversational Recommendation](https://mlsanity.com/paper/2209.03213/) (2022)
- [EventChat: Implementation and user-centric evaluation of a large language model-driven conversational recommender system for exploring leisure events in an SME context](https://mlsanity.com/paper/2407.04472/) (2024)
- [Understanding User Preferences for Interaction Styles in Conversational Recommender Systems: The Predictive Role of System Qualities, User Experience, and Traits](https://mlsanity.com/paper/2508.02328/) (2025)

## Abstract (arXiv)

Conversational recommender systems aim to interactively support online users in their information search and decision-making processes in an intuitive way. With the latest advances in voice-controlled devices, natural language processing, and AI in general, such systems received increased attention in recent years. Technically, conversational recommenders are usually complex multi-component applications and often consist of multiple machine learning models and a natural language user interface. Evaluating such a complex system in a holistic way can therefore be challenging, as it requires (i) the assessment of the quality of the different learning components, and (ii) the quality perception of the system as a whole by users. Thus, a mixed methods approach is often required, which may combine objective (computational) and subjective (perception-oriented) evaluation techniques. In this paper, we review common evaluation approaches for conversational recommender systems, identify possible limitations, and outline future directions towards more holistic evaluation practices.