# Optimizing Audio Recommendations for the Long-Term: A Reinforcement Learning Perspective

arXiv: https://arxiv.org/abs/2302.03561 · HTML: https://mlsanity.com/paper/2302.03561/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 89 references that are in the map.
- First posted on arXiv 7 Feb 2023.

## Builds on (most cited first, 18 of 18)

- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Reinforcement learning based recommender systems: A survey](https://mlsanity.com/paper/2101.06286/) (2021)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Deep Reinforcement Learning for Page-wise Recommendations](https://mlsanity.com/paper/1805.02343/) (2018)
- [A Survey on Session-based Recommender Systems](https://mlsanity.com/paper/1902.04864/) (2019)
- [RecSim: A Configurable Simulation Platform for Recommender Systems](https://mlsanity.com/paper/1909.04847/) (2019)
- [Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems](https://mlsanity.com/paper/1902.05570/) (2019)
- [Generative Adversarial User Model for Reinforcement Learning Based Recommendation System](https://mlsanity.com/paper/1812.10613/) (2018)
- [Reinforcement Learning, Bit by Bit](https://mlsanity.com/paper/2103.04047/) (2021)
- [Self-Supervised Reinforcement Learning for Recommender Systems](https://mlsanity.com/paper/2006.05779/) (2020)
- [Reinforcement Learning for Slate-based Recommender Systems: A Tractable Decomposition and Practical Methodology](https://mlsanity.com/paper/1905.12767/) (2019)
- [Targeting for long-term outcomes](https://mlsanity.com/paper/2010.15835/) (2020)
- [Environment Reconstruction with Hidden Confounders for Reinforcement Learning based Recommendation](https://mlsanity.com/paper/1907.06584/) (2019)
- [Modeling Attrition in Recommender Systems with Departing Bandits](https://mlsanity.com/paper/2203.13423/) (2022)
- [Fatigue-aware Bandits for Dependent Click Models](https://mlsanity.com/paper/2008.09733/) (2020)
- [On the Statistical Benefits of Temporal Difference Learning](https://mlsanity.com/paper/2301.13289/) (2023)
- [Approximation Benefits of Policy Gradient Methods with Aggregated States](https://mlsanity.com/paper/2007.11684/) (2020)

## Built on by (most cited first, 8 of 8)

- [ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning](https://mlsanity.com/paper/2410.09486/) (2024)
- [Impatient Bandits: Optimizing Recommendations for the Long-Term Without Delay](https://mlsanity.com/paper/2307.09943/) (2023)
- [RLocator: Reinforcement Learning for Bug Localization](https://mlsanity.com/paper/2305.05586/) (2023)
- [Optimizing Long-term Value for Auction-Based Recommender Systems via On-Policy Reinforcement Learning](https://mlsanity.com/paper/2305.13747/) (2023)
- [Regularization and Variance-Weighted Regression Achieves Minimax Optimality in Linear MDPs: Theory and Practice](https://mlsanity.com/paper/2305.13185/) (2023)
- [Value-aware Importance Weighting for Off-policy Reinforcement Learning](https://mlsanity.com/paper/2306.15625/) (2023)
- [On the Limited Representational Power of Value Functions and its Links to Statistical (In)Efficiency](https://mlsanity.com/paper/2403.07136/) (2024)
- [MultiScale Contextual Bandits for Long Term Objectives](https://mlsanity.com/paper/2503.17674/) (2025)

## Abstract (arXiv)

We present a novel podcast recommender system deployed at industrial scale. This system successfully optimizes personal listening journeys that unfold over months for hundreds of millions of listeners. In deviating from the pervasive industry practice of optimizing machine learning algorithms for short-term proxy metrics, the system substantially improves long-term performance in A/B tests. The paper offers insights into how our methods cope with attribution, coordination, and measurement challenges that usually hinder such long-term optimization. To contextualize these practical insights within a broader academic framework, we turn to reinforcement learning (RL). Using the language of RL, we formulate a comprehensive model of users' recurring relationships with a recommender system. Then, within this model, we identify our approach as a policy improvement update to a component of the existing recommender system, enhanced by tailored modeling of value functions and user-state representations. Illustrative offline experiments suggest this specialized modeling reduces data requirements by as much as a factor of 120,000 compared to black-box approaches.