# Automatic Music Playlist Generation via Simulation-based Reinforcement Learning

arXiv: https://arxiv.org/abs/2310.09123 · HTML: https://mlsanity.com/paper/2310.09123/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 34 references that are in the map.
- First posted on arXiv 13 Oct 2023.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Contextual Markov Decision Processes](https://mlsanity.com/paper/1502.02259/) (2015)
- [Reinforcement learning based recommender systems: A survey](https://mlsanity.com/paper/2101.06286/) (2021)
- [Action-Conditional Video Prediction using Deep Networks in Atari Games](https://mlsanity.com/paper/1507.08750/) (2015)
- [From Pixels to Torques: Policy Learning with Deep Dynamical Models](https://mlsanity.com/paper/1502.02251/) (2015)
- [Reinforcement Learning for Slate-based Recommender Systems: A Tractable Decomposition and Practical Methodology](https://mlsanity.com/paper/1905.12767/) (2019)
- [Exploration in Interactive Personalized Music Recommendation: A Reinforcement Learning Approach](https://mlsanity.com/paper/1311.6355/) (2013)
- [The Music Streaming Sessions Dataset](https://mlsanity.com/paper/1901.09851/) (2018)
- [Deep Reinforcement Learning with Attention for Slate Markov Decision Processes with High-Dimensional States and Actions](https://mlsanity.com/paper/1512.01124/) (2015)
- [DJ-MC: A Reinforcement-Learning Agent for Music Playlist Recommendation](https://mlsanity.com/paper/1401.1880/) (2014)
- [Towards Playlist Generation Algorithms Using RNNs Trained on Within-Track Transitions](https://mlsanity.com/paper/1606.02096/) (2016)
- [Automatic, Personalized, and Flexible Playlist Generation using Reinforcement Learning](https://mlsanity.com/paper/1809.04214/) (2018)

## Built on by (most cited first, 4 of 4)

- [Algorithmic Collective Action in Recommender Systems: Promoting Songs by Reordering Playlists](https://mlsanity.com/paper/2404.04269/) (2024)
- [Prompt-to-Slate: Diffusion Models for Prompt-Conditioned Slate Generation](https://mlsanity.com/paper/2408.06883/) (2024)
- [Exploring Large Action Sets with Hyperspherical Embeddings using von Mises-Fisher Sampling](https://mlsanity.com/paper/2507.00518/) (2025)
- [Pruning-Based Extraction of Descriptions from Probabilistic Circuits](https://mlsanity.com/paper/2311.13379/) (2023)

## Abstract (arXiv)

Personalization of playlists is a common feature in music streaming services, but conventional techniques, such as collaborative filtering, rely on explicit assumptions regarding content quality to learn how to make recommendations. Such assumptions often result in misalignment between offline model objectives and online user satisfaction metrics. In this paper, we present a reinforcement learning framework that solves for such limitations by directly optimizing for user satisfaction metrics via the use of a simulated playlist-generation environment. Using this simulator we develop and train a modified Deep Q-Network, the action head DQN (AH-DQN), in a manner that addresses the challenges imposed by the large state and action space of our RL formulation. The resulting policy is capable of making recommendations from large and dynamic sets of candidate items with the expectation of maximizing consumption metrics. We analyze and evaluate agents offline via simulations that use environment models trained on both public and proprietary streaming datasets. We show how these agents lead to better user-satisfaction metrics compared to baseline methods during online A/B tests. Finally, we demonstrate that performance assessments produced from our simulator are strongly correlated with observed online metric results.