mlsanityData to 14 Oct 2026

Constrained Reinforcement Learning for Short Video Recommendation

arXiv 2205.13248 ↗First posted 26 May 2022cs.LG cs.IRcited by 2

Not on the map. Why?

Abstract (arXiv)

The wide popularity of short videos on social media poses new opportunities and challenges to optimize recommender systems on the video-sharing platforms. Users provide complex and multi-faceted responses towards recommendations, including watch time and various types of interactions with videos. As a result, established recommendation algorithms that concern a single objective are not adequate to meet this new demand of optimizing comprehensive user experiences. In this paper, we formulate the problem of short video recommendation as a constrained Markov Decision Process (MDP), where platforms want to optimize the main goal of user watch time in long term, with the constraint of accommodating the auxiliary responses of user interactions such as sharing/downloading videos. To solve the constrained MDP, we propose a two-stage reinforcement learning approach based on actor-critic framework. At stage one, we learn individual policies to optimize each auxiliary response. At stage two, we learn a policy to (i) optimize the main response and (ii) stay close to policies learned at the first stage, which effectively guarantees the performance of this main policy on the auxiliaries. Through extensive simulations, we demonstrate effectiveness of our approach over alternatives in both optimizing the main goal as well as balancing the others. We further show the advantage of our approach in live experiments of short video recommendations, where it significantly outperforms other baselines in terms of watch time and interactions from video views. Our approach has been fully launched in the production system to optimize user experiences on the platform.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 27 of its 47 references that are in the map. First posted on arXiv 26 May 2022.

Builds on

27 of 47 references.

References · 27

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  6. 2016Wide & Deep Learning for Recommender Systemscited by 747
  7. 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 676
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  9. 2018Reward Constrained Policy Optimizationcited by 417
  10. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  11. 2018Safe Exploration in Continuous Action Spacescited by 247
  12. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 198
15 more
  1. 2021Reinforcement learning based recommender systems: A surveycited by 188
  2. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 182
  3. 2020Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Predictioncited by 177
  4. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
  5. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 145
  6. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 141
  7. 2019Reinforcement Knowledge Graph Reasoning for Explainable Recommendationcited by 102
  8. 2019Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systemscited by 90
  9. 2016Multi-Objective Deep Reinforcement Learningcited by 83
  10. 2021Towards Long-term Fairness in Recommendationcited by 77
  11. 2018Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learningcited by 71
  12. 2017Deep Reinforcement Learning for List-wise Recommendationscited by 35
  13. 2018Large-scale Interactive Recommendation with Tree-structured Policy Gradientcited by 30
  14. 2018A Multi-Objective Deep Reinforcement Learning Frameworkcited by 23
  15. 2021Choosing the Best of Both Worlds: Diverse and Novel Recommendations through Multi-Objective Reinforcement Learningcited by 14

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Goal-Conditioned Supervised Learning for Multi-Objective Recommendationcited by 1
  2. 2025Research on Short-Video Platform User Decision-Making via Multimodal Temporal Modeling and Reinforcement Learningcited by 0