# Impatient Bandits: Optimizing Recommendations for the Long-Term Without Delay

arXiv: https://arxiv.org/abs/2307.09943 · HTML: https://mlsanity.com/paper/2307.09943/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 48 references that are in the map.
- First posted on arXiv 19 Jul 2023.

## Builds on (most cited first, 17 of 17)

- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [Introduction to Multi-Armed Bandits](https://mlsanity.com/paper/1904.07272/) (2019)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [An Information-Theoretic Analysis of Thompson Sampling](https://mlsanity.com/paper/1403.5341/) (2014)
- [Online Learning under Delayed Feedback](https://mlsanity.com/paper/1306.0686/) (2013)
- [Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems](https://mlsanity.com/paper/1902.05570/) (2019)
- [Bayesian decision-making under misspecified priors with applications to meta-learning](https://mlsanity.com/paper/2107.01509/) (2021)
- [Meta Dynamic Pricing: Transfer Learning Across Experiments](https://mlsanity.com/paper/1902.10918/) (2019)
- [No Regrets for Learning the Prior in Bandits](https://mlsanity.com/paper/2107.06196/) (2021)
- [Best arm identification in multi-armed bandits with delayed feedback](https://mlsanity.com/paper/1803.10937/) (2018)
- [Deep reinforcement learning for search, recommendation, and online advertising: a survey](https://mlsanity.com/paper/1812.07127/) (2018)
- [Targeting for long-term outcomes](https://mlsanity.com/paper/2010.15835/) (2020)
- [Adaptive Experimentation in the Presence of Exogenous Nonstationary Variation](https://mlsanity.com/paper/2202.09036/) (2022)
- [Optimizing Audio Recommendations for the Long-Term: A Reinforcement Learning Perspective](https://mlsanity.com/paper/2302.03561/) (2023)
- [Partial Likelihood Thompson Sampling](https://mlsanity.com/paper/2203.00820/) (2022)
- [Hierarchical Modeling and Shrinkage for User Session Length Prediction in Media Streaming](https://mlsanity.com/paper/1803.01440/) (2018)

## Built on by (most cited first, 17 of 17)

- [Neural Contextual Bandits for Personalized Recommendation](https://mlsanity.com/paper/2312.14037/) (2023)
- [Long-term Off-Policy Evaluation and Learning](https://mlsanity.com/paper/2404.15691/) (2024)
- [Meta Clustering of Neural Bandits](https://mlsanity.com/paper/2408.05586/) (2024)
- [Addressing Missing Data Issue for Diffusion-based Recommendation](https://mlsanity.com/paper/2505.12283/) (2025)
- [Martingale Posterior Neural Networks for Fast Sequential Decision Making](https://mlsanity.com/paper/2506.11898/) (2025)
- [Don't Waste It: Guiding Generative Recommenders with Structured Human Priors via Multi-Head Decoding](https://mlsanity.com/paper/2511.10492/) (2025)
- [System-2 Recommenders: Disentangling Utility and Engagement in Recommendation Systems via Temporal Point-Processes](https://mlsanity.com/paper/2406.01611/) (2024)
- [Contextual Bandit with Herding Effects: Algorithms and Recommendation Applications](https://mlsanity.com/paper/2408.14432/) (2024)
- [Lipschitz Bandits with Stochastic Delayed Feedback](https://mlsanity.com/paper/2510.00309/) (2025)
- [Maximizing Success Rate of Payment Routing using Non-stationary Bandits](https://mlsanity.com/paper/2308.01028/) (2023)
- [Short-Long Policy Evaluation with Novel Actions](https://mlsanity.com/paper/2407.03674/) (2024)
- [Predicting Long Term Sequential Policy Value Using Softer Surrogates](https://mlsanity.com/paper/2412.20638/) (2024)
- [MultiScale Contextual Bandits for Long Term Objectives](https://mlsanity.com/paper/2503.17674/) (2025)
- [Beyond Match Maximization and Fairness: Retention-Optimized Two-Sided Matching](https://mlsanity.com/paper/2602.15752/) (2026)
- [SIGMA: A Semantic-Grounded Instruction-Driven Generative Multi-Task Recommender at AliExpress](https://mlsanity.com/paper/2602.22913/) (2026)
- [Orthogonal Learner for Estimating Heterogeneous Long-Term Treatment Effects](https://mlsanity.com/paper/2604.00915/) (2026)
- [DMRL: Document-Mediated Reinforcement Learning for Skill Optimization in Advertising Recommendation](https://mlsanity.com/paper/2609.02170/) (2026)

## Abstract (arXiv)

Recommender systems are a ubiquitous feature of online platforms. Increasingly, they are explicitly tasked with increasing users' long-term satisfaction. In this context, we study a content exploration task, which we formalize as a multi-armed bandit problem with delayed rewards. We observe that there is an apparent trade-off in choosing the learning signal: Waiting for the full reward to become available might take several weeks, hurting the rate at which learning happens, whereas measuring short-term proxy rewards reflects the actual long-term goal only imperfectly. We address this challenge in two steps. First, we develop a predictive model of delayed rewards that incorporates all information obtained to date. Full observations as well as partial (short or medium-term) outcomes are combined through a Bayesian filter to obtain a probabilistic belief. Second, we devise a bandit algorithm that takes advantage of this new predictive model. The algorithm quickly learns to identify content aligned with long-term success by carefully balancing exploration and exploitation. We apply our approach to a podcast recommendation problem, where we seek to identify shows that users engage with repeatedly over two months. We empirically validate that our approach results in substantially better performance compared to approaches that either optimize for short-term proxies, or wait for the long-term outcome to be fully realized.