mlsanityData to 7 Oct 2026

Towards Off-Policy Reinforcement Learning for Ranking Policies with Human Feedback

arXiv 2401.08959 ↗First posted 17 Jan 2024cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Probabilistic learning to rank (LTR) has been the dominating approach for optimizing the ranking metric, but cannot maximize long-term rewards. Reinforcement learning models have been proposed to maximize user long-term rewards by formulating the recommendation as a sequential decision-making problem, but could only achieve inferior accuracy compared to LTR counterparts, primarily due to the lack of online interactions and the characteristics of ranking. In this paper, we propose a new off-policy value ranking (VR) algorithm that can simultaneously maximize user long-term rewards and optimize the ranking metric offline for improved sample efficiency in a unified Expectation-Maximization (EM) framework. We theoretically and empirically show that the EM process guides the leaned policy to enjoy the benefit of integration of the future reward and ranking metric, and learn without any online interactions. Extensive offline and online experiments demonstrate the effectiveness of our methods.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 22 of its 36 references that are in the map. First posted on arXiv 17 Jan 2024.

Builds on

22 of 36 references.

References · 22

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2016Variational Inference: A Review for Statisticianscited by 1,466
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2018Self-Attentive Sequential Recommendationcited by 1,169
  9. 2015Prioritized Experience Replaycited by 1,070
  10. 2015Session-based Recommendations with Recurrent Neural Networkscited by 685
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  12. 2018Personalized Top-N Sequential Recommendation via Convolutional Sequence Embeddingcited by 473
10 more
  1. 2018Variational Autoencoders for Collaborative Filteringcited by 308
  2. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  3. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 181
  4. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 99
  5. 2019RecSim: A Configurable Simulation Platform for Recommender Systemscited by 91
  6. 2018Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learningcited by 71
  7. 2020Self-Supervised Reinforcement Learning for Recommender Systemscited by 49
  8. 2020Interactive Recommender System via Knowledge Graph-enhanced Reinforcement Learningcited by 40
  9. 2023Learning How to Propagate Messages in Graph Neural Networkscited by 23
  10. 2023A General Offline Reinforcement Learning Framework for Interactive Recommendationcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2026ELVA: Exploring Ranking-Driven Universal Multimodal Retrievalcited by 5
  2. 2024Training Verifiably Robust Agents Using Set-Based Reinforcement Learningcited by 0