# Multi-Objective Recommendation via Multivariate Policy Learning

arXiv: https://arxiv.org/abs/2405.02141 · HTML: https://mlsanity.com/paper/2405.02141/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 67 references that are in the map.
- First posted on arXiv 3 May 2024.

## Builds on (most cited first, 17 of 17)

- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [Offline A/B testing for Recommender Systems](https://mlsanity.com/paper/1801.07030/) (2018)
- [Policy Evaluation and Optimization with Continuous Treatments](https://mlsanity.com/paper/1802.06037/) (2018)
- [Multi-objective Bandits: Optimizing the Generalized Gini Index](https://mlsanity.com/paper/1706.04933/) (2017)
- [Carousel Personalization in Music Streaming Apps with Contextual Bandits](https://mlsanity.com/paper/2009.06546/) (2020)
- [Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systems](https://mlsanity.com/paper/2208.04560/) (2022)
- [BLOB : A Probabilistic Model for Recommendation that Combines Organic and Bandit Signals](https://mlsanity.com/paper/2008.12504/) (2020)
- [Long-Term Value of Exploration: Measurements, Findings and Algorithms](https://mlsanity.com/paper/2305.07764/) (2023)
- [On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n Recommendation](https://mlsanity.com/paper/2307.15053/) (2023)
- [Ad-load Balancing via Off-policy Learning in a Content Marketplace](https://mlsanity.com/paper/2309.11518/) (2023)
- [Practical Bandits: An Industry Perspective](https://mlsanity.com/paper/2302.01223/) (2023)
- [Choosing the Right Weights: Balancing Value, Strategy, and Noise in Recommender Systems](https://mlsanity.com/paper/2305.17428/) (2023)
- [Variance Reduction in Ratio Metrics for Efficient Online Experiments](https://mlsanity.com/paper/2401.04062/) (2024)
- [Learning Metrics that Maximise Power for Accelerated A/B-Tests](https://mlsanity.com/paper/2402.03915/) (2024)
- [Let's Get It Started: Fostering the Discoverability of New Releases on Deezer](https://mlsanity.com/paper/2401.02827/) (2024)
- [Mostra: A Flexible Balancing Framework to Trade-off User, Artist and Platform Objectives for Music Sequencing](https://mlsanity.com/paper/2204.10463/) (2022)

## Built on by (most cited first, 10 of 10)

- [Delta-rm OPE: Off-Policy Estimation with Pairs of Policies](https://mlsanity.com/paper/2405.10024/) (2024)
- [Additive Control Variates Dominate Self-Normalisation in Off-Policy Evaluation](https://mlsanity.com/paper/2602.14914/) (2026)
- [Multi-Metric Adaptive Experimental Design Under a Fixed Budget with Validation](https://mlsanity.com/paper/2506.03062/) (2025)
- [MultiScale Contextual Bandits for Long Term Objectives](https://mlsanity.com/paper/2503.17674/) (2025)
- [Counterfactual Inference under Thompson Sampling](https://mlsanity.com/paper/2504.08773/) (2025)
- [Meta Off-Policy Estimation](https://mlsanity.com/paper/2508.07914/) (2025)
- [CASP: Support-Aware Offline Policy Selection for Two-Stage Recommender Systems](https://mlsanity.com/paper/2604.23022/) (2026)
- [A Production-Ready RL Framework for Personalized Utility Tuning with Pareto Sweeping in Pinterest Recommender Systems](https://mlsanity.com/paper/2605.16344/) (2026)
- [Multi-Objective Ranking for Live-Streaming: Balancing Fresh and Delayed Signals with Segment-Aware Targeting](https://mlsanity.com/paper/2608.04455/) (2026)
- [When Offline Evaluation Misleads: A Diagnostic Protocol for Reward and Policy Selection in Delayed-Feedback Contextual Bandits](https://mlsanity.com/paper/2608.11560/) (2026)

## Abstract (arXiv)

Real-world recommender systems often need to balance multiple objectives when deciding which recommendations to present to users. These include behavioural signals (e.g. clicks, shares, dwell time), as well as broader objectives (e.g. diversity, fairness). Scalarisation methods are commonly used to handle this balancing task, where a weighted average of per-objective reward signals determines the final score used for ranking. Naturally, how these weights are computed exactly, is key to success for any online platform. We frame this as a decision-making task, where the scalarisation weights are actions taken to maximise an overall North Star reward (e.g. long-term user retention or growth). We extend existing policy learning methods to the continuous multivariate action domain, proposing to maximise a pessimistic lower bound on the North Star reward that the learnt policy will yield. Typical lower bounds based on normal approximations suffer from insufficient coverage, and we propose an efficient and effective policy-dependent correction for this. We provide guidance to design stochastic data collection policies, as well as highly sensitive reward signals. Empirical observations from simulations, offline and online experiments highlight the efficacy of our deployed approach.