# Reinforcing User Retention in a Billion Scale Short Video Recommender System

arXiv: https://arxiv.org/abs/2302.01724 · HTML: https://mlsanity.com/paper/2302.01724/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 3 Feb 2023.

## Builds on (most cited first, 20 of 20)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Reinforcement learning based recommender systems: A survey](https://mlsanity.com/paper/2101.06286/) (2021)
- [Reinforcement Knowledge Graph Reasoning for Explainable Recommendation](https://mlsanity.com/paper/1906.05237/) (2019)
- [Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems](https://mlsanity.com/paper/1902.05570/) (2019)
- [Towards Long-term Fairness in Recommendation](https://mlsanity.com/paper/2101.03584/) (2021)
- [KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos](https://mlsanity.com/paper/2208.08696/) (2022)
- [Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning](https://mlsanity.com/paper/1802.06501/) (2018)
- [CIRS: Bursting Filter Bubbles by Counterfactual Interactive Recommender System](https://mlsanity.com/paper/2204.01266/) (2022)
- [Deconfounding Duration Bias in Watch-time Prediction for Video Recommendation](https://mlsanity.com/paper/2206.06003/) (2022)
- [Deep Reinforcement Learning for List-wise Recommendations](https://mlsanity.com/paper/1801.00209/) (2017)
- [Large-scale Interactive Recommendation with Tree-structured Policy Gradient](https://mlsanity.com/paper/1811.05869/) (2018)
- [Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systems](https://mlsanity.com/paper/2208.04560/) (2022)
- [Real-time Short Video Recommendation on Mobile Devices](https://mlsanity.com/paper/2208.09577/) (2022)
- [Feature-aware Diversified Re-ranking with Disentangled Representations for Relevant Recommendation](https://mlsanity.com/paper/2206.05020/) (2022)
- [ResAct: Reinforcing Long-term Engagement in Sequential Recommendation with Residual Actor](https://mlsanity.com/paper/2206.02620/) (2022)

## Built on by (most cited first, 15 of 15)

- [AI Alignment with Changing and Influenceable Reward Functions](https://mlsanity.com/paper/2405.17713/) (2024)
- [EasyRL4Rec: An Easy-to-use Library for Reinforcement Learning Based Recommender Systems](https://mlsanity.com/paper/2402.15164/) (2024)
- [AURO: Reinforcement Learning for Adaptive User Retention Optimization in Recommender Systems](https://mlsanity.com/paper/2310.03984/) (2023)
- [Learned Ranking Function: From Short-term Behavior Predictions to Long-term User Satisfaction](https://mlsanity.com/paper/2408.06512/) (2024)
- [UnifiedRL: A Reinforcement Learning Algorithm Tailored for Multi-Task Fusion in Large-Scale Recommender Systems](https://mlsanity.com/paper/2404.17589/) (2024)
- [Retentive Relevance: Capturing Long-Term User Value in Recommendation Systems](https://mlsanity.com/paper/2510.07621/) (2025)
- [GRADE: Personalized Multi-Task Fusion via Group-relative Reinforcement Learning with Adaptive Dirichlet Exploration](https://mlsanity.com/paper/2510.07919/) (2025)
- [Long-term User Engagement Optimization through Model-agnostic Downstream Rewards Learning](https://mlsanity.com/paper/2607.14192/) (2026)
- [EnhancedRL: An Enhanced-State Reinforcement Learning Algorithm for Multi-Task Fusion in Recommender Systems](https://mlsanity.com/paper/2409.11678/) (2024)
- [Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed](https://mlsanity.com/paper/2412.10381/) (2024)
- [MultiScale Contextual Bandits for Long Term Objectives](https://mlsanity.com/paper/2503.17674/) (2025)
- [Short-Form Video Recommendations with Multimodal Embeddings: Addressing Cold-Start and Bias Challenges](https://mlsanity.com/paper/2507.19346/) (2025)
- [KLAN: Kuaishou Landing-page Adaptive Navigator](https://mlsanity.com/paper/2507.23459/) (2025)
- [Scale over Preference: The Impact of AI-Generated Content on Online Content Ecology](https://mlsanity.com/paper/2604.01690/) (2026)
- [PPDL: A Real-world Industrial User Retention Ratio Forecasting Framework Integrating Physical Priors with Deep Learning](https://mlsanity.com/paper/2609.13789/) (2026)

## Abstract (arXiv)

Recently, short video platforms have achieved rapid user growth by recommending interesting content to users. The objective of the recommendation is to optimize user retention, thereby driving the growth of DAU (Daily Active Users). Retention is a long-term feedback after multiple interactions of users and the system, and it is hard to decompose retention reward to each item or a list of items. Thus traditional point-wise and list-wise models are not able to optimize retention. In this paper, we choose reinforcement learning methods to optimize the retention as they are designed to maximize the long-term performance. We formulate the problem as an infinite-horizon request-based Markov Decision Process, and our objective is to minimize the accumulated time interval of multiple sessions, which is equal to improving the app open frequency and user retention. However, current reinforcement learning algorithms can not be directly applied in this setting due to uncertainty, bias, and long delay time incurred by the properties of user retention. We propose a novel method, dubbed RLUR, to address the aforementioned challenges. Both offline and live experiments show that RLUR can significantly improve user retention. RLUR has been fully launched in Kuaishou app for a long time, and achieves consistent performance improvement on user retention and DAU.