# Multi-objective Reinforcement Learning: A Tool for Pluralistic Alignment

arXiv: https://arxiv.org/abs/2410.11221 · HTML: https://mlsanity.com/paper/2410.11221/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 30 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 16 of 16)

- [Fine-Grained Human Feedback Gives Better Rewards for Language Model Training](https://mlsanity.com/paper/2306.01693/) (2023)
- [Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts](https://mlsanity.com/paper/2406.12845/) (2024)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards](https://mlsanity.com/paper/2306.04488/) (2023)
- [A Survey of Multi-Objective Sequential Decision-Making](https://mlsanity.com/paper/1402.0590/) (2014)
- [A Roadmap to Pluralistic Alignment](https://mlsanity.com/paper/2402.05070/) (2024)
- [Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment](https://mlsanity.com/paper/2402.10207/) (2024)
- [The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models](https://mlsanity.com/paper/2404.16019/) (2024)
- [Reward (Mis)design for Autonomous Driving](https://mlsanity.com/paper/2104.13906/) (2021)
- [Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties](https://mlsanity.com/paper/2309.00779/) (2023)
- [RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation](https://mlsanity.com/paper/2405.00254/) (2024)
- [Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)](https://mlsanity.com/paper/2112.15422/) (2021)
- [Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning](https://mlsanity.com/paper/2407.15762/) (2024)
- [Mapping Social Choice Theory to RLHF](https://mlsanity.com/paper/2404.13038/) (2024)
- [MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning](https://mlsanity.com/paper/2201.00012/) (2021)
- [Welfare and Fairness in Multi-objective Reinforcement Learning](https://mlsanity.com/paper/2212.01382/) (2022)

## Built on by (most cited first, 4 of 4)

- [Cultural Palette: Pluralising Culture Alignment via Multi-agent Palette](https://mlsanity.com/paper/2412.11167/) (2024)
- [CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives](https://mlsanity.com/paper/2504.10823/) (2025)
- [AI Alignment Breaks at the Edge](https://mlsanity.com/paper/2602.20042/) (2026)
- [Overton Pluralistic Reinforcement Learning for Large Language Models](https://mlsanity.com/paper/2602.20759/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) is a valuable tool for the creation of AI systems. However it may be problematic to adequately align RL based on scalar rewards if there are multiple conflicting values or stakeholders to be considered. Over the last decade multi-objective reinforcement learning (MORL) using vector rewards has emerged as an alternative to standard, scalar RL. This paper provides an overview of the role which MORL can play in creating pluralistically-aligned AI.