mlsanityData to 14 Oct 2026

Deep Reinforcement Learning for Ranking Utility Tuning in the Ad Recommender System at Pinterest

arXiv 2509.05292 ↗First posted 5 Sep 2025cs.LGcited by 1

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

The ranking utility function in an ad recommender system, which linearly combines predictions of various business goals, plays a central role in balancing values across the platform, advertisers, and users. Traditional manual tuning, while offering simplicity and interpretability, often yields suboptimal results due to its unprincipled tuning objectives, the vast amount of parameter combinations, and its lack of personalization and adaptability to seasonality. In this work, we propose a general Deep Reinforcement Learning framework for Personalized Utility Tuning (DRL-PUT) to address the challenges of multi-objective optimization within ad recommender systems. Our key contributions include: 1) Formulating the problem as a reinforcement learning task: given the state of an ad request, we predict the optimal hyperparameters to maximize a pre-defined reward. 2) Developing an approach to directly learn an optimal policy model using online serving logs, avoiding the need to estimate a value function, which is inherently challenging due to the high variance and unbalanced distribution of immediate rewards. We evaluated DRL-PUT through an online A/B experiment in Pinterest's ad recommender system. Compared to the baseline manual utility tuning approach, DRL-PUT improved the click-through rate by 9.7% and the long click-through rate by 7.7% on the treated segment. We conducted a detailed ablation study on the impact of different reward definitions and analyzed the personalization aspect of the learned policy model.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 24 of its 41 references that are in the map. First posted on arXiv 5 Sep 2025.

Builds on

24 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025RecoMind: A Reinforcement Learning Framework for Optimizing In-Session User Satisfaction in Recommendation Systems31 Jul

Other references · 23

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  2. 2016Wide & Deep Learning for Recommender Systemscited by 747
  3. 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 676
  4. 2017Deep Interest Network for Click-Through Rate Predictioncited by 544
  5. 2017Deep & Cross Network for Ad Click Predictionscited by 393
  6. 2024Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendationscited by 188
  7. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 100
  8. 2017Real-Time Bidding by Reinforcement Learning in Display Advertisingcited by 91
  9. 2018Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertisingcited by 65
  10. 2018RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertisingcited by 65
  11. 2018Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertisingcited by 61
  12. 2019DEAR: Deep Reinforcement Learning for Online Advertising Impression in Recommender Systemscited by 44
11 more
  1. 2018Deep Reinforcement Learning for Sponsored Search Real-time Biddingcited by 34
  2. 2018Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Applicationcited by 34
  3. 2022Monolith: Real Time Recommendation System With Collisionless Embedding Tablecited by 30
  4. 2020Jointly Learning to Recommend and Advertisecited by 27
  5. 2018Learning to Collaborate: Multi-Scenario Ranking via Multi-Agent Reinforcement Learningcited by 23
  6. 2018Deep reinforcement learning for search, recommendation, and online advertising: a surveycited by 23
  7. 2015Visual Search at Pinterestcited by 19
  8. 2022Rethinking Personalized Ranking at Pinterest: An End-to-End Approachcited by 16
  9. 2022Deep Page-Level Interest Network in Reinforcement Learning for Ads Allocationcited by 8
  10. 2023Towards Fairness in Personalized Ads Using Impression Variance Aware Reinforcement Learningcited by 6
  11. 2023RLTP: Reinforcement Learning to Pace for Delayed Impression Modeling in Preloaded Adscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026A Production-Ready RL Framework for Personalized Utility Tuning with Pareto Sweeping in Pinterest Recommender Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DMRL: Document-Mediated Reinforcement Learning for Skill Optimization in Advertising Recommendation
  2. 2025Multi-task Offline Reinforcement Learning for Online Advertising in Recommender Systems
  3. 2023Robust Reinforcement Learning Objectives for Sequential Recommender Systems
  4. 2025Time-Constrained Recommendations: Reinforcement Learning Strategies for E-Commerce
  5. 2024Enhancing Prediction Models with Reinforcement Learning
  6. 2025Fine-Tuning Diffusion-Based Recommender Systems via Reinforcement Learning with Reward Function Optimization
  7. 2025Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems
  8. 2022PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagement