# PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagement

arXiv: https://arxiv.org/abs/2212.02779 · HTML: https://mlsanity.com/paper/2212.02779/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 58 references that are in the map.
- First posted on arXiv 6 Dec 2022.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.04717/) (2016)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [More Robust Doubly Robust Off-policy Evaluation](https://mlsanity.com/paper/1802.03493/) (2018)
- [Sequential Recommender Systems: Challenges, Progress and Prospects](https://mlsanity.com/paper/2001.04830/) (2019)
- [SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning](https://mlsanity.com/paper/2203.10050/) (2022)
- [Offline A/B testing for Recommender Systems](https://mlsanity.com/paper/1801.07030/) (2018)
- [Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems](https://mlsanity.com/paper/1902.05570/) (2019)
- [Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning](https://mlsanity.com/paper/1805.10000/) (2018)
- [Deconfounding Duration Bias in Watch-time Prediction for Video Recommendation](https://mlsanity.com/paper/2206.06003/) (2022)
- [Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systems](https://mlsanity.com/paper/2208.04560/) (2022)
- [ResAct: Reinforcing Long-term Engagement in Sequential Recommendation with Residual Actor](https://mlsanity.com/paper/2206.02620/) (2022)
- [Reinforcement Learning to Optimize Lifetime Value in Cold-Start Recommendation](https://mlsanity.com/paper/2108.09141/) (2021)
- [Maximizing Cumulative User Engagement in Sequential Recommendation: An Online Optimization Perspective](https://mlsanity.com/paper/2006.04520/) (2020)
- [Reward Shaping for User Satisfaction in a REINFORCE Recommender](https://mlsanity.com/paper/2209.15166/) (2022)

## Built on by (most cited first, 21 of 21)

- [Reward Models in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2506.15421/) (2025)
- [Reinforcement Learning from Diverse Human Preferences](https://mlsanity.com/paper/2301.11774/) (2023)
- [EasyRL4Rec: An Easy-to-use Library for Reinforcement Learning Based Recommender Systems](https://mlsanity.com/paper/2402.15164/) (2024)
- [Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework](https://mlsanity.com/paper/2506.05619/) (2025)
- [The Behavioral Fabric of LLM-Powered GUI Agents: Human Values and Interaction Outcomes](https://mlsanity.com/paper/2601.16356/) (2026)
- [You Don't Bring Me Flowers: Mitigating Unwanted Recommendations Through Conformal Risk Control](https://mlsanity.com/paper/2507.16829/) (2025)
- [UnifiedRL: A Reinforcement Learning Algorithm Tailored for Multi-Task Fusion in Large-Scale Recommender Systems](https://mlsanity.com/paper/2404.17589/) (2024)
- [DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Reward](https://mlsanity.com/paper/2505.07257/) (2025)
- [Model-enhanced Contrastive Reinforcement Learning for Sequential Recommendation](https://mlsanity.com/paper/2310.16566/) (2023)
- [A Theoretical Framework for Partially Observed Reward-States in RLHF](https://mlsanity.com/paper/2402.03282/) (2024)
- [Preference Learning with Response Time: Robust Losses and Guarantees](https://mlsanity.com/paper/2505.22820/) (2025)
- [HiGR: Industrial-Scale Hierarchical Generative Slate Recommendation Framework in Tencent](https://mlsanity.com/paper/2512.24787/) (2025)
- [Long-term User Engagement Optimization through Model-agnostic Downstream Rewards Learning](https://mlsanity.com/paper/2607.14192/) (2026)
- [EnhancedRL: An Enhanced-State Reinforcement Learning Algorithm for Multi-Task Fusion in Recommender Systems](https://mlsanity.com/paper/2409.11678/) (2024)
- [Learning Quantitative Automata Modulo Theories](https://mlsanity.com/paper/2411.10601/) (2024)
- [Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed](https://mlsanity.com/paper/2412.10381/) (2024)
- [Improving LLM Interpretability and Performance via Guided Embedding Refinement for Sequential Recommendation](https://mlsanity.com/paper/2504.11658/) (2025)
- [KLAN: Kuaishou Landing-page Adaptive Navigator](https://mlsanity.com/paper/2507.23459/) (2025)
- [Co-Authoring the Self: A Human-AI Interface for Interest Reflection in Recommenders](https://mlsanity.com/paper/2510.08930/) (2025)
- [REAR: Test-time Preference Realignment through Reward Decomposition](https://mlsanity.com/paper/2606.30339/) (2026)
- [DCEO: Direct Causal Effect Optimization for Long-Term User Value Modeling in E-commerce Search](https://mlsanity.com/paper/2608.25635/) (2026)

## Abstract (arXiv)

Current advances in recommender systems have been remarkably successful in optimizing immediate engagement. However, long-term user engagement, a more desirable performance metric, remains difficult to improve. Meanwhile, recent reinforcement learning (RL) algorithms have shown their effectiveness in a variety of long-term goal optimization tasks. For this reason, RL is widely considered as a promising framework for optimizing long-term user engagement in recommendation. Though promising, the application of RL heavily relies on well-designed rewards, but designing rewards related to long-term user engagement is quite difficult. To mitigate the problem, we propose a novel paradigm, recommender systems with human preferences (or Preference-based Recommender systems), which allows RL recommender systems to learn from preferences about users historical behaviors rather than explicitly defined rewards. Such preferences are easily accessible through techniques such as crowdsourcing, as they do not require any expert knowledge. With PrefRec, we can fully exploit the advantages of RL in optimizing long-term goals, while avoiding complex reward engineering. PrefRec uses the preferences to automatically train a reward function in an end-to-end manner. The reward function is then used to generate learning signals to train the recommendation policy. Furthermore, we design an effective optimization method for PrefRec, which uses an additional value function, expectile regression and reward model pre-training to improve the performance. We conduct experiments on a variety of long-term user engagement optimization tasks. The results show that PrefRec significantly outperforms previous state-of-the-art methods in all the tasks.