mlsanityData to 14 Oct 2026

Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed

arXiv 2412.10381 ↗First posted 28 Nov 2024cs.IR cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

In the context of a short video & live stream mixed recommendation scenario, the live stream recommendation system (RS) decides whether to allocate at most one live stream into the video feed for each user request. To maximize long-term user engagement, it is crucial to determine an optimal live stream policy for accurate live stream allocation. The inappropriate live stream allocation policy can significantly affect the duration of the usage app and user retention, which ignores the long-term negative impact of live stream allocation. Recently, reinforcement learning (RL) has been widely applied in recommendation systems to capture long-term user engagement. However, traditional RL algorithms often face divergence and instability problems, which restricts the application and deployment in the large-scale industrial recommendation systems, especially in the aforementioned challenging scenario. To address these challenges, we propose a novel Supervised Learning-enhanced Multi-Group Actor Critic algorithm (SL-MGAC). Specifically, we introduce a supervised learning-enhanced actor-critic framework that incorporates variance reduction techniques, where multi-task reward learning helps restrict bootstrapping error accumulation during critic learning. Additionally, we design a multi-group state decomposition module for both actor and critic networks to reduce prediction variance and improve model stability. We also propose a novel reward function to prevent overly greedy live stream allocation. Empirically, we evaluate the SL-MGAC algorithm using offline policy evaluation (OPE) and online A/B testing. Experimental results demonstrate that the proposed method not only outperforms baseline methods under the platform-level constraints but also exhibits enhanced stability in online recommendation scenarios.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 42 of its 52 references that are in the map. First posted on arXiv 28 Nov 2024.

Builds on

42 of 52 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Layer Normalizationcited by 5,669
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2015Continuous control with deep reinforcement learningcited by 3,407
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  10. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  11. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  12. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
30 more
  1. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  6. 2017A Survey on Multi-Task Learningcited by 840
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  9. 2017Deep Interest Network for Click-Through Rate Predictioncited by 544
  10. 2018Reward Constrained Policy Optimizationcited by 417
  11. 2017Deep Reinforcement Learning: An Overviewcited by 333
  12. 2021Reinforcement learning based recommender systems: A surveycited by 188
  13. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  14. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  15. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  16. 2019Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systemscited by 90
  17. 2020Self-Supervised Reinforcement Learning for Recommender Systemscited by 49
  18. 2018Variance Reduction for Reinforcement Learning in Input-Driven Environmentscited by 30
  19. 2015How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategiescited by 27
  20. 2022Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systemscited by 22
  21. 2022PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagementcited by 21
  22. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
  23. 2021Cross DQN: Cross Deep Q Network for Ads Allocation in Feedcited by 19
  24. 2018Reward Estimation for Variance Reduction in Deep Reinforcement Learningcited by 17
  25. 2023Exploration and Regularization of the Latent Action Space in Recommendationcited by 17
  26. 2023Reinforcing User Retention in a Billion Scale Short Video Recommender Systemcited by 15
  27. 2023Generative Slate Recommendation with Reinforcement Learningcited by 13
  28. 2019Ctrl-Z: Recovering from Instability in Reinforcement Learningcited by 6
  29. 2024UNEX-RL: Reinforcing Long-Term Rewards in Multi-Stage Recommender Systems with UNidirectional EXecutioncited by 5
  30. 2022Sequential Search with Off-Policy Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024EnhancedRL: An Enhanced-State Reinforcement Learning Algorithm for Multi-Task Fusion in Recommender Systems
  2. 2026ProRL: Effective Reinforcement Learning for Proactive Recommendation via Rectified Policy Gradient Estimation
  3. 2024UnifiedRL: A Reinforcement Learning Algorithm Tailored for Multi-Task Fusion in Large-Scale Recommender Systems
  4. 2023Model-enhanced Contrastive Reinforcement Learning for Sequential Recommendation
  5. 2025Large Language Model-Enhanced Reinforcement Learning for Diverse and Novel Recommendations
  6. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning
  7. 2025Energy-Guided Diffusion Sampling for Long-Term User Behavior Prediction in Reinforcement Learning-based Recommendation
  8. 2024A Model-based Multi-Agent Personalized Short-Video Recommender System