# SlateFree: a Model-Free Decomposition for Reinforcement Learning with Slate Actions

arXiv: https://arxiv.org/abs/2209.01876 · HTML: https://mlsanity.com/paper/2209.01876/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 28 references that are in the map.
- First posted on arXiv 5 Sep 2022.

## Builds on (most cited first, 8 of 8)

- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Action Branching Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1711.08946/) (2017)
- [RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising](https://mlsanity.com/paper/1808.00720/) (2018)
- [Discrete Sequential Prediction of Continuous Actions for Deep RL](https://mlsanity.com/paper/1705.05035/) (2017)
- [Interactive Recommender System via Knowledge Graph-enhanced Reinforcement Learning](https://mlsanity.com/paper/2006.10389/) (2020)
- [Q-Learning in enormous action spaces via amortized approximate maximization](https://mlsanity.com/paper/2001.08116/) (2020)
- [Deep Reinforcement Learning with Attention for Slate Markov Decision Processes with High-Dimensional States and Actions](https://mlsanity.com/paper/1512.01124/) (2015)
- [Combining Reward and Rank Signals for Slate Recommendation](https://mlsanity.com/paper/2107.12455/) (2021)

## Abstract (arXiv)

We consider the problem of sequential recommendations, where at each step an agent proposes some slate of N distinct items to a user from a much larger catalog of size K>>N. The user has unknown preferences towards the recommendations and the agent takes sequential actions that optimise (in our case minimise) some user-related cost, with the help of Reinforcement Learning. The possible item combinations for a slate is binomKN, an enormous number rendering value iteration methods intractable. We prove that the slate-MDP can actually be decomposed using just K item-related Q functions per state, which describe the problem in a more compact and efficient way. Based on this, we propose a novel model-free SARSA and Q-learning algorithm that performs N parallel iterations per step, without any prior user knowledge. We call this method textttSlateFree, i.e. free-of-slates, and we show numerically that it converges very fast to the exact optimum for arbitrary user profiles, and that it outperforms alternatives from the literature.