# Reinforcement Learning for Slate-based Recommender Systems: A Tractable Decomposition and Practical Methodology

arXiv: https://arxiv.org/abs/1905.12767 · HTML: https://mlsanity.com/paper/1905.12767/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2019.

## Built on by (most cited first, 27 of 27)

- [Advances and Challenges in Conversational Recommender Systems: A Survey](https://mlsanity.com/paper/2101.09459/) (2021)
- [A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models](https://mlsanity.com/paper/2502.13187/) (2025)
- [Estimating and Penalizing Induced Preference Shifts in Recommender Systems](https://mlsanity.com/paper/2204.11966/) (2022)
- [Preference Dynamics Under Personalized Recommendations](https://mlsanity.com/paper/2205.13026/) (2022)
- [RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems](https://mlsanity.com/paper/2103.08057/) (2021)
- [SUBER: An RL Environment with Simulated Human Behavior for Recommender Systems](https://mlsanity.com/paper/2406.01631/) (2024)
- [Variation Control and Evaluation for Generative SlateRecommendations](https://mlsanity.com/paper/2102.13302/) (2021)
- [Non-Linear Reinforcement Learning in Large Action Spaces: Structural Conditions and Sample-efficiency of Posterior Sampling](https://mlsanity.com/paper/2203.08248/) (2022)
- [Offline Reinforcement Learning for Mobile Notifications](https://mlsanity.com/paper/2202.03867/) (2022)
- [Optimizing Audio Recommendations for the Long-Term: A Reinforcement Learning Perspective](https://mlsanity.com/paper/2302.03561/) (2023)
- [Deep Reinforcement Learning for Autonomous Cyber Defence: A Survey](https://mlsanity.com/paper/2310.07745/) (2023)
- [Should I send this notification? Optimizing push notifications decision making by modeling the future](https://mlsanity.com/paper/2202.08812/) (2022)
- [Prompt-to-Slate: Diffusion Models for Prompt-Conditioned Slate Generation](https://mlsanity.com/paper/2408.06883/) (2024)
- [Multi-objective Optimization of Notifications Using Offline Reinforcement Learning](https://mlsanity.com/paper/2207.03029/) (2022)
- [Automatic Music Playlist Generation via Simulation-based Reinforcement Learning](https://mlsanity.com/paper/2310.09123/) (2023)
- [Learning to Rank For Push Notifications Using Pairwise Expected Regret](https://mlsanity.com/paper/2201.07681/) (2022)
- [Time-Constrained Recommendations: Reinforcement Learning Strategies for E-Commerce](https://mlsanity.com/paper/2512.13726/) (2025)
- [Spend Search Where It Pays: Value-Guided Structured Sampling and Optimization for Generative Recommendation](https://mlsanity.com/paper/2602.10699/) (2026)
- [FINN.no Slates Dataset: A new Sequential Dataset Logging Interactions, allViewed Items and Click Responses/No-Click for Recommender Systems Research](https://mlsanity.com/paper/2111.03340/) (2021)
- [Digital Human Interactive Recommendation Decision-Making Based on Reinforcement Learning](https://mlsanity.com/paper/2210.10638/) (2022)
- [Pandering in a Flexible Representative Democracy](https://mlsanity.com/paper/2211.09986/) (2022)
- [Adaptive Interventions for Global Health: A Case Study of Malaria](https://mlsanity.com/paper/2303.02075/) (2023)
- [Representation Learning in Low-rank Slate-based Recommender Systems](https://mlsanity.com/paper/2309.08622/) (2023)
- [A Deep Reinforcement Learning Approach for Interactive Search with Sentence-level Feedback](https://mlsanity.com/paper/2310.03043/) (2023)
- [Lifting the Veil: Unlocking the Power of Depth in Q-learning](https://mlsanity.com/paper/2310.17915/) (2023)
- [Preserving Expert-Level Privacy in Offline Reinforcement Learning](https://mlsanity.com/paper/2411.13598/) (2024)
- [CASP: Support-Aware Offline Policy Selection for Two-Stage Recommender Systems](https://mlsanity.com/paper/2604.23022/) (2026)

## Abstract (arXiv)

Most practical recommender systems focus on estimating immediate user engagement without considering the long-term effects of recommendations on user behavior. Reinforcement learning (RL) methods offer the potential to optimize recommendations for long-term user engagement. However, since users are often presented with slates of multiple items - which may have interacting effects on user choice - methods are required to deal with the combinatorics of the RL action space. In this work, we address the challenge of making slate-based recommendations to optimize long-term value using RL. Our contributions are three-fold. (i) We develop SLATEQ, a decomposition of value-based temporal-difference and Q-learning that renders RL tractable with slates. Under mild assumptions on user choice behavior, we show that the long-term value (LTV) of a slate can be decomposed into a tractable function of its component item-wise LTVs. (ii) We outline a methodology that leverages existing myopic learning-based recommenders to quickly develop a recommender that handles LTV. (iii) We demonstrate our methods in simulation, and validate the scalability of decomposed TD-learning using SLATEQ in live experiments on YouTube.