# RecoMind: A Reinforcement Learning Framework for Optimizing In-Session User Satisfaction in Recommendation Systems

arXiv: https://arxiv.org/abs/2508.00201 · HTML: https://mlsanity.com/paper/2508.00201/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 22 of its 47 references that are in the map.
- First posted on arXiv 31 Jul 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [Neural Attentive Session-based Recommendation](https://mlsanity.com/paper/1711.04725/) (2017)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [A Survey of Graph Neural Networks for Recommender Systems: Challenges, Methods, and Directions](https://mlsanity.com/paper/2109.12843/) (2021)
- [Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation](https://mlsanity.com/paper/1807.01473/) (2018)
- [RecSim: A Configurable Simulation Platform for Recommender Systems](https://mlsanity.com/paper/1909.04847/) (2019)
- [Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems](https://mlsanity.com/paper/1902.05570/) (2019)
- [Deep Reinforcement Learning for List-wise Recommendations](https://mlsanity.com/paper/1801.00209/) (2017)
- [TransAct: Transformer-based Realtime User Action Model for Recommendation at Pinterest](https://mlsanity.com/paper/2306.00248/) (2023)
- [Deep reinforcement learning for search, recommendation, and online advertising: a survey](https://mlsanity.com/paper/1812.07127/) (2018)
- [A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions](https://mlsanity.com/paper/2109.03540/) (2021)
- [Rethinking Personalized Ranking at Pinterest: An End-to-End Approach](https://mlsanity.com/paper/2209.08435/) (2022)
- [Generative Slate Recommendation with Reinforcement Learning](https://mlsanity.com/paper/2301.08632/) (2023)
- [Partially Observable Markov Decision Process for Recommender Systems](https://mlsanity.com/paper/1608.07793/) (2016)
- [Off-policy Learning for Multiple Loggers](https://mlsanity.com/paper/1907.09652/) (2019)

## Built on by (most cited first, 3 of 3)

- [Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedback](https://mlsanity.com/paper/2602.12612/) (2026)
- [Deep Reinforcement Learning for Ranking Utility Tuning in the Ad Recommender System at Pinterest](https://mlsanity.com/paper/2509.05292/) (2025)
- [A Production-Ready RL Framework for Personalized Utility Tuning with Pareto Sweeping in Pinterest Recommender Systems](https://mlsanity.com/paper/2605.16344/) (2026)

## Abstract (arXiv)

Existing web-scale recommendation systems commonly use supervised learning methods that prioritize immediate user feedback. Although reinforcement learning (RL) offers a solution to optimize longer-term goals, such as in-session engagement, applying it at web scale is challenging due to the extremely large action space and engineering complexity. In this paper, we introduce RecoMind, a simulator-based RL framework designed for the effective optimization of session-based goals at web-scale. RecoMind leverages existing recommendation models to establish a simulation environment and to bootstrap the RL policy to optimize immediate user interactions from the outset. This method integrates well with existing industry pipelines, simplifying the training and deployment of RL policies. Additionally, RecoMind introduces a custom exploration strategy to efficiently explore web-scale action spaces with hundreds of millions of items. We evaluated RecoMind through extensive offline simulations and online A/B testing on a video streaming platform. Both methods showed that the RL policy trained using RecoMind significantly outperforms traditional supervised learning recommendation approaches in in-session user satisfaction. In online A/B tests, the RL policy increased videos watched for more than 10 seconds by 15.81% and improved session depth by 4.71% for sessions with at least 10 interactions. As a result, RecoMind presents a systematic and scalable approach for embedding RL into web-scale recommendation systems, showing great promise for optimizing session-based user satisfaction.