# Reinforcement Learning to Optimize Lifetime Value in Cold-Start Recommendation

arXiv: https://arxiv.org/abs/2108.09141 · HTML: https://mlsanity.com/paper/2108.09141/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 45 references that are in the map.
- First posted on arXiv 20 Aug 2021.

## Builds on (most cited first, 23 of 23)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Deep Learning based Recommender System: A Survey and New Perspectives](https://mlsanity.com/paper/1707.07435/) (2017)
- [A Survey of Heterogeneous Information Network Analysis](https://mlsanity.com/paper/1511.04854/) (2015)
- [Deep Reinforcement Learning for Page-wise Recommendations](https://mlsanity.com/paper/1805.02343/) (2018)
- [Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems](https://mlsanity.com/paper/1902.05570/) (2019)
- [Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning](https://mlsanity.com/paper/1802.06501/) (2018)
- [InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining](https://mlsanity.com/paper/2003.13198/) (2020)
- [Deep Reinforcement Learning for List-wise Recommendations](https://mlsanity.com/paper/1801.00209/) (2017)
- [Addressing the Item Cold-start Problem by Attribute-driven Active Learning](https://mlsanity.com/paper/1805.09023/) (2018)
- [Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation](https://mlsanity.com/paper/1911.03845/) (2019)
- [Learning to Collaborate: Multi-Scenario Ranking via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1809.06260/) (2018)
- [PEGASUS: A Policy Search Method for Large MDPs and POMDPs](https://mlsanity.com/paper/1301.3878/) (2013)
- [Cold-Start Reinforcement Learning with Softmax Policy Gradient](https://mlsanity.com/paper/1709.09346/) (2017)
- [Exact-K Recommendation via Maximal Clique Optimization](https://mlsanity.com/paper/1905.07089/) (2019)
- [Offline Meta-level Model-based Reinforcement Learning Approach for Cold-Start Recommendation](https://mlsanity.com/paper/2012.02476/) (2020)
- [Role of Matrix Factorization Model in Collaborative Filtering Algorithm: A Survey](https://mlsanity.com/paper/1503.07475/) (2015)
- [Speeding up the Metabolism in E-commerce by Reinforcement Mechanism Design](https://mlsanity.com/paper/1807.00448/) (2018)
- [Optimal Network Control in Partially-Controllable Networks](https://mlsanity.com/paper/1901.01517/) (2019)

## Built on by (most cited first, 9 of 9)

- [PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagement](https://mlsanity.com/paper/2212.02779/) (2022)
- [Recent Developments in Recommender Systems: A Survey](https://mlsanity.com/paper/2306.12680/) (2023)
- [STEER: Unified Style Transfer with Expert Reinforcement](https://mlsanity.com/paper/2311.07167/) (2023)
- [AURO: Reinforcement Learning for Adaptive User Retention Optimization in Recommender Systems](https://mlsanity.com/paper/2310.03984/) (2023)
- [Magical: Medical Lay Language Generation via Semantic Invariance and Layperson-tailored Adaptation](https://mlsanity.com/paper/2508.08730/) (2025)
- [Item Cold Start Recommendation via Adversarial Variational Auto-encoder Warm-up](https://mlsanity.com/paper/2302.14395/) (2023)
- [A Safe Exploration Strategy for Model-free Task Adaptation in Safety-constrained Grid Environments](https://mlsanity.com/paper/2408.00997/) (2024)
- [NeSHFS: Neighborhood Search with Heuristic-based Feature Selection for Click-Through Rate Prediction](https://mlsanity.com/paper/2409.08703/) (2024)
- [AIGP: An LLM-Based Framework for Long-Term Value Alignment in E-Commerce Pricing](https://mlsanity.com/paper/2606.26787/) (2026)

## Abstract (arXiv)

Recommender system plays a crucial role in modern E-commerce platform. Due to the lack of historical interactions between users and items, cold-start recommendation is a challenging problem. In order to alleviate the cold-start issue, most existing methods introduce content and contextual information as the auxiliary information. Nevertheless, these methods assume the recommended items behave steadily over time, while in a typical E-commerce scenario, items generally have very different performances throughout their life period. In such a situation, it would be beneficial to consider the long-term return from the item perspective, which is usually ignored in conventional methods. Reinforcement learning (RL) naturally fits such a long-term optimization problem, in which the recommender could identify high potential items, proactively allocate more user impressions to boost their growth, therefore improve the multi-period cumulative gains. Inspired by this idea, we model the process as a Partially Observable and Controllable Markov Decision Process (POC-MDP), and propose an actor-critic RL framework (RL-LTV) to incorporate the item lifetime values (LTV) into the recommendation. In RL-LTV, the critic studies historical trajectories of items and predict the future LTV of fresh item, while the actor suggests a score-based policy which maximizes the future LTV expectation. Scores suggested by the actor are then combined with classical ranking scores in a dual-rank framework, therefore the recommendation is balanced with the LTV consideration. Our method outperforms the strong live baseline with a relative improvement of 8.67% and 18.03% on IPV and GMV of cold-start items, on one of the largest E-commerce platform.