# Hierarchical Reinforcement Learning for Temporal Abstraction of Listwise Recommendation

arXiv: https://arxiv.org/abs/2409.07416 · HTML: https://mlsanity.com/paper/2409.07416/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 11 Sep 2024.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 22 of 22)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing](https://mlsanity.com/paper/1905.10083/) (2019)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Planning with Goal-Conditioned Policies](https://mlsanity.com/paper/1911.08453/) (2019)
- [Stochastic Neural Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1704.03012/) (2017)
- [STAN: Spatio-Temporal Attention Network for Next Location Recommendation](https://mlsanity.com/paper/2102.04095/) (2021)
- [Deep Reinforcement Learning for Page-wise Recommendations](https://mlsanity.com/paper/1805.02343/) (2018)
- [Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems](https://mlsanity.com/paper/1902.05570/) (2019)
- [Towards Long-term Fairness in Recommendation](https://mlsanity.com/paper/2101.03584/) (2021)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [Fast and Slow Learning of Recurrent Independent Mechanisms](https://mlsanity.com/paper/2105.08710/) (2021)
- [Deep Reinforcement Learning for List-wise Recommendations](https://mlsanity.com/paper/1801.00209/) (2017)
- [Device-Cloud Collaborative Learning for Recommendation](https://mlsanity.com/paper/2104.06624/) (2021)
- [FeedRec: News Feed Recommendation with Various User Feedbacks](https://mlsanity.com/paper/2102.04903/) (2021)
- [Exact-K Recommendation via Maximal Clique Optimization](https://mlsanity.com/paper/1905.07089/) (2019)
- [Aggregating E-commerce Search Results from Heterogeneous Sources via Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1902.08882/) (2019)
- [MC2-SF: Slow-Fast Learning for Mobile-Cloud Collaborative Recommendation](https://mlsanity.com/paper/2109.12314/) (2021)

## Built on by (most cited first, 1 of 1)

- [ProActor: Timing-Aware Reinforcement Learning for Proactive Task Scheduling Agents](https://mlsanity.com/paper/2605.24900/) (2026)

## Abstract (arXiv)

Modern listwise recommendation systems need to consider both long-term user perceptions and short-term interest shifts. Reinforcement learning can be applied on recommendation to study such a problem but is also subject to large search space, sparse user feedback and long interactive latency. Motivated by recent progress in hierarchical reinforcement learning, we propose a novel framework called mccHRL to provide different levels of temporal abstraction on listwise recommendation. Within the hierarchical framework, the high-level agent studies the evolution of user perception, while the low-level agent produces the item selection policy by modeling the process as a sequential decision-making problem. We argue that such framework has a well-defined decomposition of the outra-session context and the intra-session context, which are encoded by the high-level and low-level agents, respectively. To verify this argument, we implement both a simulator-based environment and an industrial dataset-based experiment. Results observe significant performance improvement by our method, compared with several well-known baselines. Data and codes have been made public.