# Exploration in Interactive Personalized Music Recommendation: A Reinforcement Learning Approach

arXiv: https://arxiv.org/abs/1311.6355 · HTML: https://mlsanity.com/paper/1311.6355/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2013.

## Built on by (most cited first, 22 of 22)

- [Interactive Recommendation Agent with Active User Commands](https://mlsanity.com/paper/2509.21317/) (2025)
- [SupervisorBot: NLP-Annotated Real-Time Recommendations of Psychotherapy Treatment Strategies with Deep Reinforcement Learning](https://mlsanity.com/paper/2208.13077/) (2022)
- [Generative Adversarial Reward Learning for Generalized Behavior Tendency Inference](https://mlsanity.com/paper/2105.00822/) (2021)
- [Psychotherapy AI Companion with Reinforcement Learning Recommendations and Interpretable Policy Dynamics](https://mlsanity.com/paper/2303.09601/) (2023)
- [Automatic Music Playlist Generation via Simulation-based Reinforcement Learning](https://mlsanity.com/paper/2310.09123/) (2023)
- [On the Unreasonable Efficiency of State Space Clustering in Personalization Tasks](https://mlsanity.com/paper/2112.13141/) (2021)
- [Examining Policy Entropy of Reinforcement Learning Agents for Personalization Tasks](https://mlsanity.com/paper/2211.11869/) (2022)
- [A Deep Reinforcement Learning Approach for Traffic Signal Control Optimization](https://mlsanity.com/paper/2107.06115/) (2021)
- [Reinforcement Learning for Linear Quadratic Control is Vulnerable Under Cost Manipulation](https://mlsanity.com/paper/2203.05774/) (2022)
- [Simulated Contextual Bandits for Personalization Tasks from Recommendation Datasets](https://mlsanity.com/paper/2210.10631/) (2022)
- [Optimization of Epsilon-Greedy Exploration](https://mlsanity.com/paper/2506.03324/) (2025)
- [An Empirical Analysis on Transparent Algorithmic Exploration in Recommender Systems](https://mlsanity.com/paper/2108.00151/) (2021)
- [SINGA-Easy: An Easy-to-Use Framework for MultiModal Analysis](https://mlsanity.com/paper/2108.02572/) (2021)
- [VisualTextRank: Unsupervised Graph-based Content Extraction for Automating Ad Text to Image Search](https://mlsanity.com/paper/2108.02725/) (2021)
- [Text is NOT Enough: Integrating Visual Impressions into Open-domain Dialogue Generation](https://mlsanity.com/paper/2109.05778/) (2021)
- [Discovering Personalized Semantics for Soft Attributes in Recommender Systems using Concept Activation Vectors](https://mlsanity.com/paper/2202.02830/) (2022)
- [Reinforced MOOCs Concept Recommendation in Heterogeneous Information Networks](https://mlsanity.com/paper/2203.11011/) (2022)
- [Explanation-Guided Fairness Testing through Genetic Algorithm](https://mlsanity.com/paper/2205.08335/) (2022)
- [Using Graph Neural Networks for Program Termination](https://mlsanity.com/paper/2207.14648/) (2022)
- [Tribe or Not? Critical Inspection of Group Differences Using TribalGram](https://mlsanity.com/paper/2303.09664/) (2023)
- [Increasing Entropy to Boost Policy Gradient Performance on Personalization Tasks](https://mlsanity.com/paper/2310.05324/) (2023)
- [Breaking the Cold-Start Barrier: Reinforcement Learning with Double and Dueling DQNs](https://mlsanity.com/paper/2508.21259/) (2025)

## Abstract (arXiv)

Current music recommender systems typically act in a greedy fashion by recommending songs with the highest user ratings. Greedy recommendation, however, is suboptimal over the long term: it does not actively gather information on user preferences and fails to recommend novel songs that are potentially interesting. A successful recommender system must balance the needs to explore user preferences and to exploit this information for recommendation. This paper presents a new approach to music recommendation by formulating this exploration-exploitation trade-off as a reinforcement learning task called the multi-armed bandit. To learn user preferences, it uses a Bayesian model, which accounts for both audio content and the novelty of recommendations. A piecewise-linear approximation to the model and a variational inference algorithm are employed to speed up Bayesian inference. One additional benefit of our approach is a single unified model for both music recommendation and playlist generation. Both simulation results and a user study indicate strong potential for the new approach.