mlsanityData to 7 Oct 2026

Behavior Alignment: A New Perspective of Evaluating LLM-based Conversational Recommender Systems

arXiv 2404.11773 ↗First posted 17 Apr 2024cs.IR cs.AIcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated great potential in Conversational Recommender Systems (CRS). However, the application of LLMs to CRS has exposed a notable discrepancy in behavior between LLM-based CRS and human recommenders: LLMs often appear inflexible and passive, frequently rushing to complete the recommendation task without sufficient inquiry.This behavior discrepancy can lead to decreased accuracy in recommendations and lower user satisfaction. Despite its importance, existing studies in CRS lack a study about how to measure such behavior discrepancy. To fill this gap, we propose Behavior Alignment, a new evaluation metric to measure how well the recommendation strategies made by a LLM-based CRS are consistent with human recommenders'. Our experiment results show that the new metric is better aligned with human preferences and can better differentiate how systems perform than existing evaluation metrics. As Behavior Alignment requires explicit and costly human annotations on the recommendation strategies, we also propose a classification-based method to implicitly measure the Behavior Alignment based on the responses. The evaluation results confirm the robustness of the method.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 26 references that are in the map. First posted on arXiv 17 Apr 2024.

Builds on

19 of 26 references.

References · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  3. 2023TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendationcited by 336
  4. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 307
  5. 2018Towards Deep Conversational Recommendationscited by 174
  6. 2023Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approachcited by 168
  7. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  8. 2020Improving Conversational Recommender Systems via Knowledge Graph based Semantic Fusioncited by 122
  9. 2021Advances and Challenges in Conversational Recommender Systems: A Surveycited by 118
  10. 2018Conversational Recommender Systemcited by 115
  11. 2019Towards Knowledge-Based Recommender Dialog Systemcited by 112
  12. 2020Towards Topic-Guided Conversational Recommender Systemcited by 94
7 more
  1. 2020INSPIRED: Toward Sociable Recommendation Dialog Systemscited by 91
  2. 2022Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learningcited by 85
  3. 2023Leveraging Large Language Models in Conversational Recommender Systemscited by 63
  4. 2020Evaluating Conversational Recommender Systems via User Simulationcited by 46
  5. 2021CRSLab: An Open-Source Toolkit for Building Conversational Recommender Systemcited by 34
  6. 2021Zero-Shot Recommendation as Language Modelingcited by 28
  7. 2022Evaluating Conversational Recommender Systems: A Landscape of Researchcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 3.

11 papers.

  1. 2025Adaptive Multi-Subspace Representation Steering for Attribute Alignment in Large Language Modelscited by 16
  2. 2026Global Evolutionary Steering: Refining Activation Steering Control via Cross-Layer Consistencycited by 4
  3. 2025The Pitfalls of Growing Group Complexity: LLMs and Social Choice-Based Aggregation for Group Recommendationscited by 3
  4. 2025GGBond: Growing Graph-Based AI-Agent Society for Socially-Aware Recommender Simulationcited by 3
  5. 2025PerFairX: Is There a Balance Between Fairness and Personality in Large Language Model Recommendations?cited by 3
  6. 2026Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testingcited by 3
  7. 2025Consistent Explainers or Unreliable Narrators? Understanding LLM-generated Group Recommendationscited by 2
  8. 2025ReGeS: Reciprocal Retrieval-Generation Synergy for Conversational Recommender Systemscited by 2
  9. 2025Event Segmentation Applications in Large Language Model Enabled Automated Recall Assessmentscited by 1
  10. 2025USB-Rec: An Effective Framework for Improving Conversational Recommendation Capability of Large Language Modelcited by 0
  11. 2026Consensus vs. Dissent: Dynamic LLM Modeling of Subjective Preferences in Group Recommenderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Conversational Recommender Systems via Large Language Models: A User-Centric Framework
  2. 2024Aligning Large Language Models for Controllable Recommendations
  3. 2023Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models
  4. 2023RecExplainer: Aligning Large Language Models for Explaining Recommendation Models
  5. 2023How Can Recommender Systems Benefit from Large Language Models: A Survey
  6. 2023A Large Language Model Enhanced Conversational Recommender System
  7. 2026Toward User Preference Alignment in LLM Recommendation via Explicit Context Feedback
  8. 2025RLHF Fine-Tuning of LLMs for Alignment with Implicit User Feedback in Conversational Recommenders