# Large-scale Interactive Recommendation with Tree-structured Policy Gradient

arXiv: https://arxiv.org/abs/1811.05869 · HTML: https://mlsanity.com/paper/1811.05869/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2018.

## Built on by (most cited first, 30 of 30)

- [Advances and Challenges in Conversational Recommender Systems: A Survey](https://mlsanity.com/paper/2101.09459/) (2021)
- [Multi-Task Recommendations with Reinforcement Learning](https://mlsanity.com/paper/2302.03328/) (2023)
- [A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions](https://mlsanity.com/paper/2109.03540/) (2021)
- [Breaking the Linear Iteration Cost Barrier for Some Well-known Conditional Gradient Methods Using MaxIP Data-structures](https://mlsanity.com/paper/2111.15139/) (2021)
- [Two-Stage Constrained Actor-Critic for Short Video Recommendation](https://mlsanity.com/paper/2302.01680/) (2023)
- [Reinforcing User Retention in a Billion Scale Short Video Recommender System](https://mlsanity.com/paper/2302.01724/) (2023)
- [Towards Return Parity in Markov Decision Processes](https://mlsanity.com/paper/2111.10476/) (2021)
- [Causal Decision Transformer for Recommender Systems via Offline Reinforcement Learning](https://mlsanity.com/paper/2304.07920/) (2023)
- [Conversational Tree Search: A New Hybrid Dialog Task](https://mlsanity.com/paper/2303.10227/) (2023)
- [Generative Adversarial Reward Learning for Generalized Behavior Tendency Inference](https://mlsanity.com/paper/2105.00822/) (2021)
- [SAPIENT: Mastering Multi-turn Conversational Recommendation with Strategic Planning and Monte Carlo Tree Search](https://mlsanity.com/paper/2410.09580/) (2024)
- [AURO: Reinforcement Learning for Adaptive User Retention Optimization in Recommender Systems](https://mlsanity.com/paper/2310.03984/) (2023)
- [Reinforcement Learning based Path Exploration for Sequential Explainable Recommendation](https://mlsanity.com/paper/2111.12262/) (2021)
- [Retentive Decision Transformer with Adaptive Masking for Reinforcement Learning based Recommendation Systems](https://mlsanity.com/paper/2403.17634/) (2024)
- [Constrained Reinforcement Learning for Short Video Recommendation](https://mlsanity.com/paper/2205.13248/) (2022)
- [Learning from Imperfect Human Feedback: a Tale from Corruption-Robust Dueling](https://mlsanity.com/paper/2405.11204/) (2024)
- [DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Reward](https://mlsanity.com/paper/2505.07257/) (2025)
- [Conditional Sequential Slate Optimization](https://mlsanity.com/paper/2108.05618/) (2021)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)
- [Reinforcement Learning Guided Multi-Objective Exam Paper Generation](https://mlsanity.com/paper/2303.01042/) (2023)
- [AutoAssign+: Automatic Shared Embedding Assignment in Streaming Recommendation](https://mlsanity.com/paper/2308.06965/) (2023)
- [On Causally Disentangled State Representation Learning for Reinforcement Learning based Recommender Systems](https://mlsanity.com/paper/2407.13091/) (2024)
- [RPAF: A Reinforcement Prediction-Allocation Framework for Cache Allocation in Large-Scale Recommender Systems](https://mlsanity.com/paper/2409.13175/) (2024)
- [The Importance of Causality in Decision Making: A Perspective on Recommender Systems](https://mlsanity.com/paper/2410.01822/) (2024)
- [Adversarial Robustness of Deep Reinforcement Learning based Dynamic Recommender Systems](https://mlsanity.com/paper/2112.00973/) (2021)
- [Who Are the Best Adopters? User Selection Model for Free Trial Item Promotion](https://mlsanity.com/paper/2202.09508/) (2022)
- [Towards Validating Long-Term User Feedbacks in Interactive Recommendation Systems](https://mlsanity.com/paper/2308.11137/) (2023)
- [KLAN: Kuaishou Landing-page Adaptive Navigator](https://mlsanity.com/paper/2507.23459/) (2025)
- [Fine-Tuning Diffusion-Based Recommender Systems via Reinforcement Learning with Reward Function Optimization](https://mlsanity.com/paper/2511.06937/) (2025)
- [ZeroCode: On-demand Error-Correcting Code Construction from the Zero Matrix via Reinforcement Learning](https://mlsanity.com/paper/2609.34265/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) has recently been introduced to interactive recommender systems (IRS) because of its nature of learning from dynamic interactions and planning for long-run performance. As IRS is always with thousands of items to recommend (i.e., thousands of actions), most existing RL-based methods, however, fail to handle such a large discrete action space problem and thus become inefficient. The existing work that tries to deal with the large discrete action space problem by utilizing the deep deterministic policy gradient framework suffers from the inconsistency between the continuous action representation (the output of the actor network) and the real discrete action. To avoid such inconsistency and achieve high efficiency and recommendation effectiveness, in this paper, we propose a Tree-structured Policy Gradient Recommendation (TPGR) framework, where a balanced hierarchical clustering tree is built over the items and picking an item is formulated as seeking a path from the root to a certain leaf of the tree. Extensive experiments on carefully-designed environments based on two real-world datasets demonstrate that our model provides superior recommendation performance and significant efficiency improvement over state-of-the-art methods.