# A Production-Ready RL Framework for Personalized Utility Tuning with Pareto Sweeping in Pinterest Recommender Systems

arXiv: https://arxiv.org/abs/2605.16344 · HTML: https://mlsanity.com/paper/2605.16344/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 8 May 2026.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [Deep Reinforcement Learning for Page-wise Recommendations](https://mlsanity.com/paper/1805.02343/) (2018)
- [Real-Time Bidding by Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1701.02490/) (2017)
- [Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1610.02707/) (2016)
- [Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1802.09756/) (2018)
- [Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1802.08365/) (2018)
- [DEAR: Deep Reinforcement Learning for Online Advertising Impression in Recommender Systems](https://mlsanity.com/paper/1909.03602/) (2019)
- [Deep Reinforcement Learning for Sponsored Search Real-time Bidding](https://mlsanity.com/paper/1803.00259/) (2018)
- [Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application](https://mlsanity.com/paper/1803.00710/) (2018)
- [Monolith: Real Time Recommendation System With Collisionless Embedding Table](https://mlsanity.com/paper/2209.07663/) (2022)
- [Deep reinforcement learning for search, recommendation, and online advertising: a survey](https://mlsanity.com/paper/1812.07127/) (2018)
- [Controllable Multi-Objective Re-ranking with Policy Hypernetworks](https://mlsanity.com/paper/2306.05118/) (2023)
- [Multi-Objective Recommendation via Multivariate Policy Learning](https://mlsanity.com/paper/2405.02141/) (2024)
- [Choosing the Right Weights: Balancing Value, Strategy, and Noise in Recommender Systems](https://mlsanity.com/paper/2305.17428/) (2023)
- [RLTP: Reinforcement Learning to Pace for Delayed Impression Modeling in Preloaded Ads](https://mlsanity.com/paper/2302.02592/) (2023)
- [RecoMind: A Reinforcement Learning Framework for Optimizing In-Session User Satisfaction in Recommendation Systems](https://mlsanity.com/paper/2508.00201/) (2025)
- [A Hybrid Meta-Learning and Multi-Armed Bandit Approach for Context-Specific Multi-Objective Recommendation Optimization](https://mlsanity.com/paper/2409.08752/) (2024)
- [Deep Reinforcement Learning for Ranking Utility Tuning in the Ad Recommender System at Pinterest](https://mlsanity.com/paper/2509.05292/) (2025)

## Abstract (arXiv)

Large-scale recommenders encode multi-objective trade-offs by combining multiple predicted outcomes into a single utility score. Although this utility layer can be updated independently of the ranker, weight tuning remains largely manual, globally applied, slow to adapt to changing environments and business needs, and hard to govern as priorities shift. We propose PRL-PUTS, a Production-ready, ranker independent RL framework for Personalized Utility-weight Tuning with Pareto Sweeping. We cast utility tuning as a one-step, value-based RL problem: given request context, an agent selects a utility-weight vector that re-weights ranker predictions to maximize request-level engagement rewards. To visualize performance across the trade-off spectrum and allow decision makers to update the deployed operating policy instantly, we adopt an inference-time Pareto frontier sweeping via a scalarization parameter, producing a family of policies and an empirical Pareto frontier used as a governance artifact for operating policy selection. PRL-PUTS runs in parallel with ranking inference without adding serving latency. We validate PRL-PUTS with offline analysis using unbiased exploration logs and online experiments on Pinterest Homefeed where PRL-PUTS showed significant increases in engagement compared to baseline such as +0.13% increase in successful session, a core metric for user engagement.