mlsanityData to 7 Oct 2026

Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systems

arXiv 2208.04560 ↗First posted 9 Aug 2022cs.IR cs.AIcited by 22

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Recommender System (RS) is an important online application that affects billions of users every day. The mainstream RS ranking framework is composed of two parts: a Multi-Task Learning model (MTL) that predicts various user feedback, i.e., clicks, likes, sharings, and a Multi-Task Fusion model (MTF) that combines the multi-task outputs into one final ranking score with respect to user satisfaction. There has not been much research on the fusion model while it has great impact on the final recommendation as the last crucial process of the ranking. To optimize long-term user satisfaction rather than obtain instant returns greedily, we formulate MTF task as Markov Decision Process (MDP) within a recommendation session and propose a Batch Reinforcement Learning (RL) based Multi-Task Fusion framework (BatchRL-MTF) that includes a Batch RL framework and an online exploration. The former exploits Batch RL to learn an optimal recommendation policy from the fixed batch data offline for long-term user satisfaction, while the latter explores potential high-value actions online to break through the local optimal dilemma. With a comprehensive investigation on user behaviors, we model the user satisfaction reward with subtle heuristics from two aspects of user stickiness and user activeness. Finally, we conduct extensive experiments on a billion-sample level real-world dataset to show the effectiveness of our model. We propose a conservative offline policy estimator (Conservative-OPEstimator) to test our model offline. Furthermore, we take online experiments in a real recommendation environment to compare performance of different models. As one of few Batch RL researches applied in MTF task successfully, our model has also been deployed on a large-scale industrial short video platform, serving hundreds of millions of users.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 38 references that are in the map. First posted on arXiv 9 Aug 2022.

Builds on

17 of 38 references.

References · 17

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2017Deep Interest Network for Click-Through Rate Predictioncited by 541
  6. 2018Deep Interest Evolution Network for Click-Through Rate Predictioncited by 336
  7. 2019Batch Policy Learning under Constraintscited by 242
  8. 2021Reinforcement learning based recommender systems: A surveycited by 187
  9. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 158
  10. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  11. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  12. 2019Practice on Long Sequential User Behavior Modeling for Click-Through Rate Predictioncited by 118
5 more
  1. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  2. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 99
  3. 2019Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systemscited by 90
  4. 2019Deep Spatio-Temporal Neural Networks for Click-Through Rate Predictioncited by 17
  5. 2019Value-aware Recommendation based on Reinforced Profit Maximization in E-commerce Systemscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 11; 2025: 2; 2026: 2.

22 papers.

Sort:

By area:

  1. 2023Multi-Task Recommendations with Reinforcement Learningcited by 26
  2. 2022PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagementcited by 21
  3. 2023Reinforcing User Retention in a Billion Scale Short Video Recommender Systemcited by 15
  4. 2024Multi-Objective Recommendation via Multivariate Policy Learningcited by 10
  5. 2024EasyRL4Rec: An Easy-to-use Library for Reinforcement Learning Based Recommender Systemscited by 8
  6. 2024TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agentscited by 7
  7. 2024M4: Multi-Proxy Multi-Gate Mixture of Experts Network for Multiple Instance Learning in Histopathology Image Analysiscited by 6
  8. 2023Sim2Rec: A Simulator-based Decision-making Approach to Optimize Real-World Long-term User Engagement in Sequential Recommender Systemscited by 5
  9. 2023AURO: Reinforcement Learning for Adaptive User Retention Optimization in Recommender Systemscited by 5
  10. 2024Learned Ranking Function: From Short-term Behavior Predictions to Long-term User Satisfactioncited by 4
  11. 2024UnifiedRL: A Reinforcement Learning Algorithm Tailored for Multi-Task Fusion in Large-Scale Recommender Systemscited by 2
  12. 2024Dynamic User Interest Augmentation via Stream Clustering and Memory Networks in Large-Scale Recommender Systemscited by 2
10 more
  1. 2023Hierarchical Reinforcement Learning for Modeling User Novelty-Seeking Intent in Recommender Systemscited by 1
  2. 2023AutoAssign+: Automatic Shared Embedding Assignment in Streaming Recommendationcited by 1
  3. 2025GRADE: Personalized Multi-Task Fusion via Group-relative Reinforcement Learning with Adaptive Dirichlet Explorationcited by 1
  4. 2025PrivORL: Differentially Private Synthetic Dataset for Offline Reinforcement Learningcited by 1
  5. 2026Long-term User Engagement Optimization through Model-agnostic Downstream Rewards Learningcited by 1
  6. 2024Aligning GPTRec with Beyond-Accuracy Goals with Reinforcement Learningcited by 0
  7. 2024Advancements in Recommender Systems: A Comprehensive Analysis Based on Data, Algorithms, and Evaluationcited by 0
  8. 2024EnhancedRL: An Enhanced-State Reinforcement Learning Algorithm for Multi-Task Fusion in Recommender Systemscited by 0
  9. 2024Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feedcited by 0
  10. 2026CALMRec: Causally Aligned Language Memory for Long-Horizon Recommendationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems
  2. 2023Towards Validating Long-Term User Feedbacks in Interactive Recommendation Systems
  3. 2020Offline Meta-level Model-based Reinforcement Learning Approach for Cold-Start Recommendation
  4. 2025MTRec: Learning to Align with User Preferences via Mental Reward Models
  5. 2023Model-enhanced Contrastive Reinforcement Learning for Sequential Recommendation
  6. 2023On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systems
  7. 2026FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning
  8. 2025Fine-Tuning Diffusion-Based Recommender Systems via Reinforcement Learning with Reward Function Optimization