# Behavior Alignment: A New Perspective of Evaluating LLM-based Conversational Recommender Systems

arXiv: https://arxiv.org/abs/2404.11773 · HTML: https://mlsanity.com/paper/2404.11773/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 17 Apr 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Preference alignment for recommenders and RAG](https://mlsanity.com/alignment/preference-alignment-for-recommenders-and-rag/)

## Builds on (most cited first, 19 of 19)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation](https://mlsanity.com/paper/2305.00447/) (2023)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [Towards Deep Conversational Recommendations](https://mlsanity.com/paper/1812.07617/) (2018)
- [Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approach](https://mlsanity.com/paper/2305.07001/) (2023)
- [Survey on Evaluation Methods for Dialogue Systems](https://mlsanity.com/paper/1905.04071/) (2019)
- [Improving Conversational Recommender Systems via Knowledge Graph based Semantic Fusion](https://mlsanity.com/paper/2007.04032/) (2020)
- [Advances and Challenges in Conversational Recommender Systems: A Survey](https://mlsanity.com/paper/2101.09459/) (2021)
- [Conversational Recommender System](https://mlsanity.com/paper/1806.03277/) (2018)
- [Towards Knowledge-Based Recommender Dialog System](https://mlsanity.com/paper/1908.05391/) (2019)
- [Towards Topic-Guided Conversational Recommender System](https://mlsanity.com/paper/2010.04125/) (2020)
- [INSPIRED: Toward Sociable Recommendation Dialog Systems](https://mlsanity.com/paper/2009.14306/) (2020)
- [Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning](https://mlsanity.com/paper/2206.09363/) (2022)
- [Leveraging Large Language Models in Conversational Recommender Systems](https://mlsanity.com/paper/2305.07961/) (2023)
- [Evaluating Conversational Recommender Systems via User Simulation](https://mlsanity.com/paper/2006.08732/) (2020)
- [CRSLab: An Open-Source Toolkit for Building Conversational Recommender System](https://mlsanity.com/paper/2101.00939/) (2021)
- [Zero-Shot Recommendation as Language Modeling](https://mlsanity.com/paper/2112.04184/) (2021)
- [Evaluating Conversational Recommender Systems: A Landscape of Research](https://mlsanity.com/paper/2208.12061/) (2022)

## Built on by (most cited first, 11 of 11)

- [Adaptive Multi-Subspace Representation Steering for Attribute Alignment in Large Language Models](https://mlsanity.com/paper/2508.10599/) (2025)
- [Global Evolutionary Steering: Refining Activation Steering Control via Cross-Layer Consistency](https://mlsanity.com/paper/2603.12298/) (2026)
- [Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testing](https://mlsanity.com/paper/2601.04554/) (2026)
- [The Pitfalls of Growing Group Complexity: LLMs and Social Choice-Based Aggregation for Group Recommendations](https://mlsanity.com/paper/2505.05016/) (2025)
- [GGBond: Growing Graph-Based AI-Agent Society for Socially-Aware Recommender Simulation](https://mlsanity.com/paper/2505.21154/) (2025)
- [PerFairX: Is There a Balance Between Fairness and Personality in Large Language Model Recommendations?](https://mlsanity.com/paper/2509.08829/) (2025)
- [Consistent Explainers or Unreliable Narrators? Understanding LLM-generated Group Recommendations](https://mlsanity.com/paper/2507.13705/) (2025)
- [ReGeS: Reciprocal Retrieval-Generation Synergy for Conversational Recommender Systems](https://mlsanity.com/paper/2509.21371/) (2025)
- [Event Segmentation Applications in Large Language Model Enabled Automated Recall Assessments](https://mlsanity.com/paper/2502.13349/) (2025)
- [USB-Rec: An Effective Framework for Improving Conversational Recommendation Capability of Large Language Model](https://mlsanity.com/paper/2509.20381/) (2025)
- [Consensus vs. Dissent: Dynamic LLM Modeling of Subjective Preferences in Group Recommenders](https://mlsanity.com/paper/2607.10235/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated great potential in Conversational Recommender Systems (CRS). However, the application of LLMs to CRS has exposed a notable discrepancy in behavior between LLM-based CRS and human recommenders: LLMs often appear inflexible and passive, frequently rushing to complete the recommendation task without sufficient inquiry.This behavior discrepancy can lead to decreased accuracy in recommendations and lower user satisfaction. Despite its importance, existing studies in CRS lack a study about how to measure such behavior discrepancy. To fill this gap, we propose Behavior Alignment, a new evaluation metric to measure how well the recommendation strategies made by a LLM-based CRS are consistent with human recommenders'. Our experiment results show that the new metric is better aligned with human preferences and can better differentiate how systems perform than existing evaluation metrics. As Behavior Alignment requires explicit and costly human annotations on the recommendation strategies, we also propose a classification-based method to implicitly measure the Behavior Alignment based on the responses. The evaluation results confirm the robustness of the method.