mlsanityData to 7 Oct 2026

Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models

arXiv 2305.13112 ↗First posted 22 May 2023cs.CL cs.IRcited by 48

On the map: LLM reasoning → LLM reasoning for recommendation

Abstract (arXiv)

The recent success of large language models (LLMs) has shown great potential to develop more powerful conversational recommender systems (CRSs), which rely on natural language conversations to satisfy user needs. In this paper, we embark on an investigation into the utilization of ChatGPT for conversational recommendation, revealing the inadequacy of the existing evaluation protocol. It might over-emphasize the matching with the ground-truth items or utterances generated by human annotators, while neglecting the interactive nature of being a capable CRS. To overcome the limitation, we further propose an interactive Evaluation approach based on LLMs named iEvaLM that harnesses LLM-based user simulators. Our evaluation approach can simulate various interaction scenarios between users and systems. Through the experiments on two publicly available CRS datasets, we demonstrate notable improvements compared to the prevailing evaluation protocol. Furthermore, we emphasize the evaluation of explainability, and ChatGPT showcases persuasive explanation generation for its recommendations. Our study contributes to a deeper comprehension of the untapped potential of LLMs for CRSs and provides a more flexible and easy-to-use evaluation framework for future research endeavors. The codes and data are publicly available at https://github.com/RUCAIBox/iEvaLM-CRS.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in LLM reasoning for recommendation, one of 17 directions in LLM reasoning. Builds on 37 of its 50 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

37 of 50 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  5. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  6. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  7. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  8. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 307
  9. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  10. 2018Towards Deep Conversational Recommendationscited by 174
  11. 2023Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approachcited by 168
  12. 2018Neural Approaches to Conversational AIcited by 164
25 more
  1. 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
  2. 2023Uncovering ChatGPT's Capabilities in Recommender Systemscited by 160
  3. 2020A Survey on Conversational Recommender Systemscited by 142
  4. 2020Improving Conversational Recommender Systems via Knowledge Graph based Semantic Fusioncited by 122
  5. 2021Advances and Challenges in Conversational Recommender Systems: A Surveycited by 118
  6. 2023Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Predictioncited by 113
  7. 2019Towards Knowledge-Based Recommender Dialog Systemcited by 112
  8. 2020Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systemscited by 98
  9. 2022Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learningcited by 85
  10. 2023Large Language Models are Diverse Role-Players for Summarization Evaluationcited by 62
  11. 2021A Survey on Accuracy-oriented Neural Recommendation: From Collaborative Filtering to Information-rich Recommendationcited by 59
  12. 2023Is ChatGPT Equipped with Emotional Dialogue Capabilities?cited by 54
  13. 2020Evaluating Conversational Recommender Systems via User Simulationcited by 46
  14. 2020What does BERT know about books, movies and music? Probing BERT for Conversational Recommendationcited by 36
  15. 2021CRSLab: An Open-Source Toolkit for Building Conversational Recommender Systemcited by 34
  16. 2022A Unified Multi-task Learning Framework for Multi-goal Conversational Recommender Systemscited by 32
  17. 2023A Preliminary Evaluation of ChatGPT for Zero-shot Dialogue Understandingcited by 31
  18. 2021Improving Conversational Recommendation Systems' Quality with Context-Aware Item Meta Informationcited by 26
  19. 2022Evaluating Mixed-initiative Conversational Search Systems via User Simulationcited by 24
  20. 2022Variational Reasoning over Incomplete Knowledge Graphs for Conversational Recommendationcited by 22
  21. 2022BARCOR: Towards A Unified Framework for Conversational Recommendation Systemscited by 21
  22. 2021RecInDial: A Unified Framework for Conversational Recommendation with Pretrained Language Modelscited by 14
  23. 2022Measuring "Why" in Recommender Systems: a Comprehensive Survey on the Evaluation of Explainable Recommendationcited by 14
  24. 2023UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systemscited by 11
  25. 2022Aligning Recommendation and Conversation via Dual Imitationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 12; 2025: 12; 2026: 9.

48 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Models for Recommendationcited by 378
  2. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  3. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 188
  4. 2023Recommender AI Agent: Integrating Large Language Models for Interactive Recommendationscited by 130
  5. 2023Large Language Models as Zero-Shot Conversational Recommenderscited by 126
  6. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 94
  7. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  8. 2023Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agentscited by 76
  9. 2023Large Language Models for Generative Recommendation: A Survey and Visionary Discussionscited by 72
  10. 2023Amazon-M2: A Multilingual Multi-locale Shopping Session Dataset for Recommendation and Text Generationcited by 42
  11. 2024LLM-Powered User Simulator for Recommender Systemcited by 38
  12. 2023A Large Language Model Enhanced Conversational Recommender Systemcited by 31
36 more
  1. 2024How Reliable is Your Simulator? Analysis on the Limitations of Current LLM-based User Simulators for Conversational Recommendationcited by 25
  2. 2024A Survey of Conversational Searchcited by 22
  3. 2023MuseChat: A Conversational Music Recommendation System for Videoscited by 21
  4. 2023User Modeling in the Era of Large Language Models: Current Research and Future Directionscited by 18
  5. 2025A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigmscited by 17
  6. 2024A LLM-based Controllable, Scalable, Human-Involved User Simulator Framework for Conversational Recommender Systemscited by 12
  7. 2024Exploring the Impact of Large Language Models on Recommender Systems: An Extensive Reviewcited by 11
  8. 2025RecUserSim: A Realistic and Diverse User Simulator for Evaluating Conversational Recommender Systemscited by 11
  9. 2023Improving Conversational Recommendation Systems via Counterfactual Data Simulationcited by 10
  10. 2024Concept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factorscited by 9
  11. 2024Towards Empathetic Conversational Recommender Systemscited by 9
  12. 2025Human vs. Agent in Task-Oriented Conversationscited by 9
  13. 2024Mitigating Propensity Bias of Large Language Models for Recommender Systemscited by 8
  14. 2023A New Dialogue Response Generation Agent for Large Language Models by Asking Questions to Detect User's Intentionscited by 6
  15. 2024Bridging Conversational and Collaborative Signals for Conversational Recommendationcited by 5
  16. 2025Search-Based Interaction For Conversation Recommendation via Generative Reward Model Based Simulated Usercited by 5
  17. 2024Reasoning over User Preferences: Knowledge Graph-Augmented LLMs for Explainable Conversational Recommendationscited by 4
  18. 2025LaViC: Adapting Large Vision-Language Models to Visually-Aware Conversational Recommendationcited by 4
  19. 2025Reinforced Strategy Optimization for Conversational Recommender Systems via Network-of-Expertscited by 4
  20. 2024Keyword-driven Retrieval-Augmented Large Language Models for Cold-start User Recommendationscited by 3
  21. 2025Can LLMs Outshine Conventional Recommenders? A Comparative Evaluationcited by 3
  22. 2026Towards Secure Retrieval-Augmented Generation: A Comprehensive Review of Threats, Defenses and Benchmarkscited by 3
  23. 2026GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verificationcited by 3
  24. 2023EALM: Introducing Multidimensional Ethical Alignment in Conversational Information Retrievalcited by 2
  25. 2025RefuteBench 2.0 -- Agentic Benchmark for Dynamic Evaluation of LLM Responses to Refutation Instructioncited by 2
  26. 2026τ-Rec: A Verifiable Benchmark for Agentic Recommender Systemscited by 2
  27. 2026Whom Do AI Agents Work For? Role Assignment Induces Sponsorship Bias in LLM Recommenderscited by 2
  28. 2025Multi-Type Context-Aware Conversational Recommender Systems via Mixture-of-Expertscited by 1
  29. 2025A Language-Driven Framework for Improving Personalized Recommendations: Merging LLMs with Traditional Algorithmscited by 0
  30. 2025Enhancing Serendipity Recommendation System by Constructing Dynamic User Knowledge Graphs with Large Language Modelscited by 0
  31. 2025USB-Rec: An Effective Framework for Improving Conversational Recommendation Capability of Large Language Modelcited by 0
  32. 2026CoReflect: A Reflective Co-Evolution Framework for Improving Conversational Evaluationcited by 0
  33. 2026Interplay: Training Independent Simulators for Reference-Free Conversational Recommendationcited by 0
  34. 2026User Simulator-Guided Multi-Turn Preference Optimization for Reasoning LLM-based Conversational Recommendationcited by 0
  35. 2026Controllable User Simulationcited by 0
  36. 2026AURA: Agentic Diagnosis and Refinement for Production Recommender Systems at Scalecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Conversational Recommender Systems via Large Language Models: A User-Centric Framework
  2. 2024Behavior Alignment: A New Perspective of Evaluating LLM-based Conversational Recommender Systems
  3. 2025Graph Retrieval-Augmented LLM for Conversational Recommendation Systems
  4. 2024EventChat: Implementation and user-centric evaluation of a large language model-driven conversational recommender system for exploring leisure events in an SME context
  5. 2024Large Language Model Driven Recommendation
  6. 2025A Framework for Generating Conversational Recommendation Datasets from Behavioral Interactions
  7. 2025Collaborative Retrieval for Large Language Model-based Conversational Recommender Systems
  8. 2025A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challenges