mlsanityData to 7 Oct 2026

Evaluating Conversational Recommender Systems via User Simulation

arXiv 2006.08732 ↗First posted 15 Jun 2020cs.IRcited by 46

Not on the map. Why?

Abstract (arXiv)

Conversational information access is an emerging research area. Currently, human evaluation is used for end-to-end system evaluation, which is both very time and resource intensive at scale, and thus becomes a bottleneck of progress. As an alternative, we propose automated evaluation by means of simulating users. Our user simulator aims to generate responses that a real human would give by considering both individual preferences and the general flow of interaction with the system. We evaluate our simulation approach on an item recommendation task by comparing three existing conversational recommender systems. We show that preference modeling and task-specific interaction models both contribute to more realistic simulations, and can help achieve high correlation between automatic evaluation measures and manual human assessments.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 4; 2023: 12; 2024: 9; 2025: 8; 2026: 9.

46 papers.

Sort:

By area:

  1. 2023User Behavior Simulation with Large Language Model based Agentscited by 136
  2. 2021Advances and Challenges in Conversational Recommender Systems: A Surveycited by 118
  3. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  4. 2023Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactionscited by 70
  5. 2023Leveraging Large Language Models in Conversational Recommender Systemscited by 63
  6. 2023Rethinking Model Evaluation as Narrowing the Socio-Technical Gapcited by 56
  7. 2023Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Modelscited by 48
  8. 2024Evaluating Large Language Models as Generative User Simulators for Conversational Recommendationcited by 45
  9. 2022Neural Approaches to Conversational Information Retrievalcited by 41
  10. 2023Knowledge-enhanced Mixed-initiative Dialogue System for Emotional Support Conversationscited by 41
  11. 2023User Simulation for Evaluating Information Access Systemscited by 30
  12. 2022Evaluating Mixed-initiative Conversational Search Systems via User Simulationcited by 24
34 more
  1. 2026A Survey on LLM-based Conversational User Simulationcited by 14
  2. 2021Analysing Mixed Initiatives and Search Strategies during Conversational Searchcited by 12
  3. 2024Behavior Alignment: A New Perspective of Evaluating LLM-based Conversational Recommender Systemscited by 11
  4. 2025RecUserSim: A Realistic and Diverse User Simulator for Evaluating Conversational Recommender Systemscited by 11
  5. 2021Generating Usage-related Questions for Preference Elicitation in Conversational Recommender Systemscited by 10
  6. 2026ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenderscited by 10
  7. 2022Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systemscited by 9
  8. 2023Talk the Walk: Synthetic Data Generation for Conversational Music Recommendationcited by 9
  9. 2025Human vs. Agent in Task-Oriented Conversationscited by 9
  10. 2023Human Choice Prediction in Language-based Persuasion Games: Simulation-based Off-Policy Evaluationcited by 8
  11. 2024Incorporating External Knowledge and Goal Guidance for LLM-based Conversational Recommender Systemscited by 8
  12. 2025Simulating Before Planning: Constructing Intrinsic User World Model for User-Tailored Dialogue Policy Planningcited by 7
  13. 2025Towards Personalized Conversational Sales Agents: Contextual User Profiling for Strategic Actioncited by 6
  14. 2026Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Tracescited by 6
  15. 2021Designing an Automatic Agent for Repeated Language based Persuasion Gamescited by 5
  16. 2023Alleviating the Long-Tail Problem in Conversational Recommender Systemscited by 5
  17. 2024SAPIENT: Mastering Multi-turn Conversational Recommendation with Strategic Planning and Monte Carlo Tree Searchcited by 5
  18. 2024Multi-trait User Simulation with Adaptive Decoding for Conversational Task Assistantscited by 5
  19. 2025How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyondcited by 4
  20. 2025Reinforced Strategy Optimization for Conversational Recommender Systems via Network-of-Expertscited by 4
  21. 2023A Conversation is Worth A Thousand Recommendations: A Survey of Holistic Conversational Recommender Systemscited by 3
  22. 2025Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Frameworkcited by 3
  23. 2024Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMscited by 2
  24. 2024RecSys Arena: Pair-wise Recommender System Evaluation with Large Language Modelscited by 1
  25. 2025Fashion-AlterEval: A Dataset for Improved Evaluation of Conversational Recommendation Systems with Alternative Relevant Itemscited by 1
  26. 2026ShiJianBench: From Dialogue to Decision for Long-Horizon Evaluation of Investment Advisorscited by 1
  27. 2022Situating Recommender Systems in Practice: Towards Inductive Learning and Incremental Updatescited by 0
  28. 2023DiQAD: A Benchmark Dataset for End-to-End Open-domain Dialogue Assessmentcited by 0
  29. 2024EventChat: Implementation and user-centric evaluation of a large language model-driven conversational recommender system for exploring leisure events in an SME contextcited by 0
  30. 2026Interplay: Training Independent Simulators for Reference-Free Conversational Recommendationcited by 0
  31. 2026Decision-aware User Simulation Agent for Evaluating Conversational Recommender Systemscited by 0
  32. 2026Prompt Optimization for User Simulation in Conversational Recommender Systems: A Multi-Objective Frameworkcited by 0
  33. 2026Disclosure-Gated User Simulation for Companion-Agent Evaluationcited by 0
  34. 2026On Evaluating and Improving Conversational Agents in Productioncited by 0