# Navigating the Social Welfare Frontier: Portfolios for Multi-objective Reinforcement Learning

arXiv: https://arxiv.org/abs/2502.09724 · HTML: https://mlsanity.com/paper/2502.09724/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 13 Feb 2025.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 16 of 16)

- [A Survey of Multi-Objective Sequential Decision-Making](https://mlsanity.com/paper/1402.0590/) (2014)
- [A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation](https://mlsanity.com/paper/1908.08342/) (2019)
- [MaxMin-RLHF: Alignment with Diverse Human Preferences](https://mlsanity.com/paper/2402.08925/) (2024)
- [RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation](https://mlsanity.com/paper/2405.00254/) (2024)
- [Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization](https://mlsanity.com/paper/2301.07784/) (2023)
- [Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey](https://mlsanity.com/paper/1909.02964/) (2019)
- [Joint Trajectory and Communication Design for Multi-UAV Enabled Wireless Networks](https://mlsanity.com/paper/1705.02723/) (2017)
- [Pareto Conditioned Networks](https://mlsanity.com/paper/2204.05036/) (2022)
- [Welfare and Fairness in Multi-objective Reinforcement Learning](https://mlsanity.com/paper/2212.01382/) (2022)
- [Approximation Algorithms for Minimum Norm and Ordered Optimization Problems](https://mlsanity.com/paper/1811.05022/) (2018)
- [Which Lp norm is the fairest? Approximations for fair facility location across all "p"](https://mlsanity.com/paper/2211.14873/) (2022)
- [Balancing Act: Prioritization Strategies for LLM-Designed Restless Bandit Rewards](https://mlsanity.com/paper/2408.12112/) (2024)
- [Policy Aggregation](https://mlsanity.com/paper/2411.03651/) (2024)
- [Learning Social Welfare Functions](https://mlsanity.com/paper/2405.17700/) (2024)
- [Balancing Notions of Equity: Trade-offs Between Fair Portfolio Sizes and Achievable Guarantees](https://mlsanity.com/paper/2311.03230/) (2023)
- [Data-Driven Solution Portfolios](https://mlsanity.com/paper/2412.00717/) (2024)

## Built on by (most cited first, 5 of 5)

- [Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework](https://mlsanity.com/paper/2506.05619/) (2025)
- [Health Facility Location in Ethiopia: Leveraging LLMs to Integrate Expert Knowledge into Algorithmic Planning](https://mlsanity.com/paper/2601.11479/) (2026)
- [Many Preferences, Few Policies: Towards Scalable Language Model Personalization](https://mlsanity.com/paper/2604.04144/) (2026)
- [Multi-User Dueling Bandits: A Fair Approach using Nash Social Welfare](https://mlsanity.com/paper/2605.01961/) (2026)
- [Why Global LLM Leaderboards Are Misleading: Small Portfolios for Heterogeneous Supervised ML](https://mlsanity.com/paper/2605.06656/) (2026)

## Abstract (arXiv)

In many real-world applications of reinforcement learning (RL), deployed policies have varied impacts on different stakeholders, creating challenges in reaching consensus on how to effectively aggregate their preferences. Generalized p-means form a widely used class of social welfare functions for this purpose, with broad applications in fair resource allocation, AI alignment, and decision-making. This class includes well-known welfare functions such as Egalitarian, Nash, and Utilitarian welfare. However, selecting the appropriate social welfare function is challenging for decision-makers, as the structure and outcomes of optimal policies can be highly sensitive to the choice of p. To address this challenge, we study the concept of an α-approximate portfolio in RL, a set of policies that are approximately optimal across the family of generalized p-means for all p ∈ [-∞, 1]. We propose algorithms to compute such portfolios and provide theoretical guarantees on the trade-offs among approximation factor, portfolio size, and computational efficiency. Experimental results on synthetic and real-world datasets demonstrate the effectiveness of our approach in summarizing the policy space induced by varying p values, empowering decision-makers to navigate this landscape more effectively.