mlsanityData to 7 Oct 2026

Concept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factors

arXiv 2404.03304 ↗First posted 4 Apr 2024cs.CL cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

The conversational recommendation system (CRS) has been criticized regarding its user experience in real-world scenarios, despite recent significant progress achieved in academia. Existing evaluation protocols for CRS may prioritize system-centric factors such as effectiveness and fluency in conversation while neglecting user-centric aspects. Thus, we propose a new and inclusive evaluation protocol, Concept, which integrates both system- and user-centric factors. We conceptualise three key characteristics in representing such factors and further divide them into six primary abilities. To implement Concept, we adopt a LLM-based user simulator and evaluator with scoring rubrics that are tailored for each primary ability. Our protocol, Concept, serves a dual purpose. First, it provides an overview of the pros and cons in current CRS models. Second, it pinpoints the problem of low usability in the "omnipotent" ChatGPT and offers a comprehensive reference guide for evaluating CRS, thereby setting the foundation for CRS improvement.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 44 of its 116 references that are in the map. First posted on arXiv 4 Apr 2024.

Builds on

44 of 116 references.

References · 44

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2018Datasheets for Datasetscited by 1,371
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  6. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  7. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  8. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  9. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  10. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  11. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  12. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
32 more
  1. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  2. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  3. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  4. 2018Towards Deep Conversational Recommendationscited by 174
  5. 2018The Design and Implementation of XiaoIce, an Empathetic Social Chatbotcited by 172
  6. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
  7. 2020A Survey on Conversational Recommender Systemscited by 142
  8. 2017A Knowledge-Grounded Neural Conversation Modelcited by 136
  9. 2021Advances and Challenges in Conversational Recommender Systems: A Surveycited by 118
  10. 2018Conversational Recommender Systemcited by 115
  11. 2019Towards Knowledge-Based Recommender Dialog Systemcited by 112
  12. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  13. 2020INSPIRED: Toward Sociable Recommendation Dialog Systemscited by 91
  14. 2022Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learningcited by 85
  15. 2023Is GPT-4 a Good Data Analyst?cited by 69
  16. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 68
  17. 2019How should my chatbot interact? A survey on human-chatbot interaction designcited by 56
  18. 2023Calibrating LLM-Based Evaluatorcited by 53
  19. 2023Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Modelscited by 48
  20. 2022A Survey of Ad Hoc Teamwork Researchcited by 41
  21. 2020CR-Walker: Tree-Structured Graph Reasoning and Dialog Acts for Conversational Recommendationcited by 31
  22. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Taskscited by 23
  23. 2022Variational Reasoning over Incomplete Knowledge Graphs for Conversational Recommendationcited by 22
  24. 2022BARCOR: Towards A Unified Framework for Conversational Recommendation Systemscited by 21
  25. 2022Rank List Sensitivity of Recommender Systems to Interaction Perturbationscited by 20
  26. 2023In-Context Learning User Simulators for Task-Oriented Dialog Systemscited by 19
  27. 2023Reflective Linguistic Programming (RLP): A Stepping Stone in Socially-Aware AGI (SocialAGI)cited by 16
  28. 2023TREA: Tree-Structure Reasoning Schema for Conversational Recommendationcited by 15
  29. 2023Conversational Recommender System and Large Language Model Are Made for Each Other in E-commerce Pre-sales Dialoguecited by 15
  30. 2023UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systemscited by 11
  31. 2022Aligning Recommendation and Conversation via Dual Imitationcited by 6
  32. 2023A Conversation is Worth A Thousand Recommendations: A Survey of Holistic Conversational Recommender Systemscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 3.

9 papers.

  1. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  2. 2024CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Modelscited by 48
  3. 2025Reading Between the Prompts: How Stereotypes Shape LLM's Implicit Personalizationcited by 24
  4. 2025Towards Personalized Conversational Sales Agents: Contextual User Profiling for Strategic Actioncited by 6
  5. 2025How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyondcited by 4
  6. 2026METRO: Towards Strategy Induction from Expert Dialogue Transcripts for Non-collaborative Dialoguescited by 2
  7. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluatorscited by 1
  8. 2026Beyond Prompt: Fine-grained Simulation of Cognitively Impaired Standardized Patients via Stochastic Steeringcited by 0
  9. 2026RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commercecited by 0