mlsanityData to 14 Oct 2026

Optimizing Long-term Value for Auction-Based Recommender Systems via On-Policy Reinforcement Learning

arXiv 2305.13747 ↗First posted 23 May 2023cs.IR cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Auction-based recommender systems are prevalent in online advertising platforms, but they are typically optimized to allocate recommendation slots based on immediate expected return metrics, neglecting the downstream effects of recommendations on user behavior. In this study, we employ reinforcement learning to optimize for long-term return metrics in an auction-based recommender system. Utilizing temporal difference learning, a fundamental reinforcement learning algorithm, we implement an one-step policy improvement approach that biases the system towards recommendations with higher long-term user engagement metrics. This optimizes value over long horizons while maintaining compatibility with the auction framework. Our approach is grounded in dynamic programming ideas which show that our method provably improves upon the existing auction-based base policy. Through an online A/B test conducted on an auction-based recommender system which handles billions of impressions and users daily, we empirically establish that our proposed method outperforms the current production system in terms of long-term user engagement metrics.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 18 of its 51 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

18 of 51 references.

References · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2016Wide & Deep Learning for Recommender Systemscited by 747
  4. 2015Session-based Recommendations with Recurrent Neural Networkscited by 695
  5. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  6. 2015Contextual Markov Decision Processescited by 220
  7. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 182
  8. 2019Sequential Recommender Systems: Challenges, Progress and Prospectscited by 115
  9. 2017Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigationcited by 98
  10. 2019Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systemscited by 90
  11. 2018Generative Adversarial User Model for Reinforcement Learning Based Recommendation Systemcited by 64
  12. 2021Reinforcement Learning, Bit by Bitcited by 59
6 more
  1. 2018Sequence-Aware Recommender Systemscited by 58
  2. 2020Self-Supervised Reinforcement Learning for Recommender Systemscited by 49
  3. 2018Deep reinforcement learning for search, recommendation, and online advertising: a surveycited by 23
  4. 2021Deep Exploration for Recommendation Systemscited by 14
  5. 2023Optimizing Audio Recommendations for the Long-Term: A Reinforcement Learning Perspectivecited by 8
  6. 2023Evaluating Online Bandit Exploration In Large-Scale Recommender Systemcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 0; 2025: 1; 2026: 1.

6 papers.

  1. 2023Filter Bubbles in Recommender Systems: Fact or Fallacy -- A Systematic Reviewcited by 13
  2. 2023Scalable Neural Contextual Bandit for Recommender Systemscited by 12
  3. 2023Pearl: A Production-ready Reinforcement Learning Agentcited by 9
  4. 2023Non-Stationary Contextual Bandit Learning via Neural Predictive Ensemble Samplingcited by 1
  5. 2026Long-term User Engagement Optimization through Model-agnostic Downstream Rewards Learningcited by 1
  6. 2025Learning to Clean: Reinforcement Learning for Noisy Label Correctioncited by 0