# Unified Conversational Recommendation Policy Learning via Graph-based Reinforcement Learning

arXiv: https://arxiv.org/abs/2105.09710 · HTML: https://mlsanity.com/paper/2105.09710/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2021.

## Built on by (most cited first, 30 of 51)

- [Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration](https://mlsanity.com/paper/2305.13626/) (2023)
- [When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunities](https://mlsanity.com/paper/2307.16376/) (2023)
- [Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents](https://mlsanity.com/paper/2311.00262/) (2023)
- [A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospects](https://mlsanity.com/paper/2305.02750/) (2023)
- [Leveraging Large Language Models in Conversational Recommender Systems](https://mlsanity.com/paper/2305.07961/) (2023)
- [Towards Human-centered Proactive Conversational Agents](https://mlsanity.com/paper/2404.12670/) (2024)
- [PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Finance](https://mlsanity.com/paper/2210.08817/) (2022)
- [Knowledge-enhanced Mixed-initiative Dialogue System for Emotional Support Conversations](https://mlsanity.com/paper/2305.10172/) (2023)
- [A Unified Multi-task Learning Framework for Multi-goal Conversational Recommender Systems](https://mlsanity.com/paper/2204.06923/) (2022)
- [How Reliable is Your Simulator? Analysis on the Limitations of Current LLM-based User Simulators for Conversational Recommendation](https://mlsanity.com/paper/2403.16416/) (2024)
- [Learning Neural Templates for Recommender Dialogue System](https://mlsanity.com/paper/2109.12302/) (2021)
- [Variational Reasoning over Incomplete Knowledge Graphs for Conversational Recommendation](https://mlsanity.com/paper/2212.11868/) (2022)
- [A Survey of Conversational Search](https://mlsanity.com/paper/2410.15576/) (2024)
- [User Satisfaction Estimation with Sequential Dialogue Act Modeling in Goal-oriented Conversational Systems](https://mlsanity.com/paper/2202.02912/) (2022)
- [Reinforcement learning on graphs: A survey](https://mlsanity.com/paper/2204.06127/) (2022)
- [TREA: Tree-Structure Reasoning Schema for Conversational Recommendation](https://mlsanity.com/paper/2307.10543/) (2023)
- [RecInDial: A Unified Framework for Conversational Recommendation with Pretrained Language Models](https://mlsanity.com/paper/2110.07477/) (2021)
- [A LLM-based Controllable, Scalable, Human-Involved User Simulator Framework for Conversational Recommender Systems](https://mlsanity.com/paper/2405.08035/) (2024)
- [Bundle MCR: Towards Conversational Bundle Recommendation](https://mlsanity.com/paper/2207.12628/) (2022)
- [Hierarchical Conversational Preference Elicitation with Bandit Feedback](https://mlsanity.com/paper/2209.06129/) (2022)
- [Wizard of Shopping: Target-Oriented E-commerce Dialogue Generation with Decision Tree Branching](https://mlsanity.com/paper/2502.00969/) (2025)
- [Towards Empathetic Conversational Recommender Systems](https://mlsanity.com/paper/2409.10527/) (2024)
- [Towards Goal-oriented Intelligent Tutoring Systems in Online Education](https://mlsanity.com/paper/2312.10053/) (2023)
- [Beyond Persuasion: Towards Conversational Recommender System with Credible Explanations](https://mlsanity.com/paper/2409.14399/) (2024)
- [Meta Policy Learning for Cold-Start Conversational Recommendation](https://mlsanity.com/paper/2205.11788/) (2022)
- [Aligning Recommendation and Conversation via Dual Imitation](https://mlsanity.com/paper/2211.02848/) (2022)
- [Lending Interaction Wings to Recommender Systems with Conversational Agents](https://mlsanity.com/paper/2310.04230/) (2023)
- [SAPIENT: Mastering Multi-turn Conversational Recommendation with Strategic Planning and Monte Carlo Tree Search](https://mlsanity.com/paper/2410.09580/) (2024)
- [Multi-grained Hypergraph Interest Modeling for Conversational Recommendation](https://mlsanity.com/paper/2305.04798/) (2023)
- [Knowledge-Enhanced Conversational Recommendation via Transformer-based Sequential Modelling](https://mlsanity.com/paper/2412.02415/) (2024)

## Abstract (arXiv)

Conversational recommender systems (CRS) enable the traditional recommender systems to explicitly acquire user preferences towards items and attributes through interactive conversations. Reinforcement learning (RL) is widely adopted to learn conversational recommendation policies to decide what attributes to ask, which items to recommend, and when to ask or recommend, at each conversation turn. However, existing methods mainly target at solving one or two of these three decision-making problems in CRS with separated conversation and recommendation components, which restrict the scalability and generality of CRS and fall short of preserving a stable training procedure. In the light of these challenges, we propose to formulate these three decision-making problems in CRS as a unified policy learning task. In order to systematically integrate conversation and recommendation components, we develop a dynamic weighted graph based RL method to learn a policy to select the action at each conversation turn, either asking an attribute or recommending items. Further, to deal with the sample efficiency issue, we propose two action selection strategies for reducing the candidate action space according to the preference and entropy information. Experimental results on two benchmark CRS datasets and a real-world E-Commerce application show that the proposed method not only significantly outperforms state-of-the-art methods but also enhances the scalability and stability of CRS.