# Welfare and Fairness in Multi-objective Reinforcement Learning

arXiv: https://arxiv.org/abs/2212.01382 · HTML: https://mlsanity.com/paper/2212.01382/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 38 references that are in the map.
- First posted on arXiv 30 Nov 2022.

## Builds on (most cited first, 6 of 6)

- [A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation](https://mlsanity.com/paper/1908.08342/) (2019)
- [Fairness in Reinforcement Learning](https://mlsanity.com/paper/1611.03071/) (2016)
- [Learning Fair Policies in Multiobjective (Deep) Reinforcement Learning with Average and Discounted Rewards](https://mlsanity.com/paper/2008.07773/) (2020)
- [Fair Allocation of Indivisible Public Goods](https://mlsanity.com/paper/1805.03164/) (2018)
- [Universal and Tight Online Algorithms for Generalized-Mean Welfare](https://mlsanity.com/paper/2109.00874/) (2021)
- [APX-Hardness of Maximizing Nash Social Welfare with Indivisible Items](https://mlsanity.com/paper/1507.01159/) (2015)

## Built on by (most cited first, 20 of 20)

- [The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithm](https://mlsanity.com/paper/2406.07826/) (2024)
- [Fairness in Preference-based Reinforcement Learning](https://mlsanity.com/paper/2306.09995/) (2023)
- [Remembering to Be Fair: Non-Markovian Fairness in Sequential Decision Making](https://mlsanity.com/paper/2312.04772/) (2023)
- [Policy Aggregation](https://mlsanity.com/paper/2411.03651/) (2024)
- [Multi-objective Reinforcement Learning with Nonlinear Preferences: Provable Approximation for Maximizing Expected Scalarized Return](https://mlsanity.com/paper/2311.02544/) (2023)
- [Navigating the Social Welfare Frontier: Portfolios for Multi-objective Reinforcement Learning](https://mlsanity.com/paper/2502.09724/) (2025)
- [Multi-Objective Reinforcement Learning with Max-Min Criterion: A Game-Theoretic Approach](https://mlsanity.com/paper/2510.20235/) (2025)
- [Multi-objective Reinforcement Learning: A Tool for Pluralistic Alignment](https://mlsanity.com/paper/2410.11221/) (2024)
- [Scalable Multi-Objective Reinforcement Learning with Fairness Guarantees using Lorenz Dominance](https://mlsanity.com/paper/2411.18195/) (2024)
- [FairDICE: Fairness-Driven Offline Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2506.08062/) (2025)
- [Learning Fair Pareto-Optimal Policies in Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2606.18111/) (2026)
- [Learning in Multi-Objective Public Goods Games with Non-Linear Utilities](https://mlsanity.com/paper/2408.00682/) (2024)
- [State-of-the-art in Robot Learning for Multi-Robot Collaboration: A Comprehensive Survey](https://mlsanity.com/paper/2408.11822/) (2024)
- [Fair Resource Allocation in Weakly Coupled Markov Decision Processes](https://mlsanity.com/paper/2411.09804/) (2024)
- [[Re] FairDICE: A Fair Tradeoff in Multi-objective Offline RL](https://mlsanity.com/paper/2603.03454/) (2026)
- [Multi-User Dueling Bandits: A Fair Approach using Nash Social Welfare](https://mlsanity.com/paper/2605.01961/) (2026)
- [Constrained Multi-Objective Reinforcement Learning with Max-Min Criterion](https://mlsanity.com/paper/2605.31388/) (2026)
- [AETDICE: Unified Framework and Offline Optimization for Nonlinear Multi-Objective RL](https://mlsanity.com/paper/2606.31178/) (2026)
- [Inference-Time Policy Alignment for Fair Reinforcement Learning](https://mlsanity.com/paper/2608.00175/) (2026)
- [Fair Policy Optimization in Major-Minor Weakly Coupled Markov Decision Processes](https://mlsanity.com/paper/2609.36174/) (2026)

## Abstract (arXiv)

We study fair multi-objective reinforcement learning in which an agent must learn a policy that simultaneously achieves high reward on multiple dimensions of a vector-valued reward. Motivated by the fair resource allocation literature, we model this as an expected welfare maximization problem, for some nonlinear fair welfare function of the vector of long-term cumulative rewards. One canonical example of such a function is the Nash Social Welfare, or geometric mean, the log transform of which is also known as the Proportional Fairness objective. We show that even approximately optimal optimization of the expected Nash Social Welfare is computationally intractable even in the tabular case. Nevertheless, we provide a novel adaptation of Q-learning that combines nonlinear scalarized learning updates and non-stationary action selection to learn effective policies for optimizing nonlinear welfare functions. We show that our algorithm is provably convergent, and we demonstrate experimentally that our approach outperforms techniques based on linear scalarization, mixtures of optimal linear scalarizations, or stationary action selection for the Nash Social Welfare Objective.