# A Practical Guide to Multi-Objective Reinforcement Learning and Planning

arXiv: https://arxiv.org/abs/2103.09568 · HTML: https://mlsanity.com/paper/2103.09568/

## Key facts

- Cited by 256 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 225 references that are in the map.
- First posted on arXiv 17 Mar 2021.

## Builds on (most cited first, 30 of 33)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [A Survey of Multi-Objective Sequential Decision-Making](https://mlsanity.com/paper/1402.0590/) (2014)
- [Optimization of Molecules via Deep Reinforcement Learning](https://mlsanity.com/paper/1810.08678/) (2018)
- [A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation](https://mlsanity.com/paper/1908.08342/) (2019)
- [Universal Successor Features Approximators](https://mlsanity.com/paper/1812.07626/) (2018)
- [Dynamic Weights in Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1809.07803/) (2018)
- [Deep Reinforcement Learning for Multi-objective Optimization](https://mlsanity.com/paper/1906.02386/) (2019)
- [Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1610.02707/) (2016)
- [Learning Fair Policies in Multiobjective (Deep) Reinforcement Learning with Average and Discounted Rewards](https://mlsanity.com/paper/2008.07773/) (2020)
- [A Distributional View on Multi-Objective Policy Optimization](https://mlsanity.com/paper/2005.07513/) (2020)
- [Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey](https://mlsanity.com/paper/1909.02964/) (2019)
- [Meta-Learning for Multi-objective Reinforcement Learning](https://mlsanity.com/paper/1811.03376/) (2018)
- [Molecular Design in Synthetically Accessible Chemical Space via Deep Reinforcement Learning](https://mlsanity.com/paper/2004.14308/) (2020)
- [Urban Driving with Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1811.08586/) (2018)
- [Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning](https://mlsanity.com/paper/1903.08542/) (2019)
- [A Multi-Objective Deep Reinforcement Learning Framework](https://mlsanity.com/paper/1803.02965/) (2018)
- [Multi-objective Contextual Bandit Problem with Similarity Information](https://mlsanity.com/paper/1803.04015/) (2018)
- [Ordered Preference Elicitation Strategies for Supporting Multi-Objective Decision Making](https://mlsanity.com/paper/1802.07606/) (2018)
- [Universal Successor Representations for Transfer Reinforcement Learning](https://mlsanity.com/paper/1804.03758/) (2018)
- [A utility-based analysis of equilibria in multi-objective normal form games](https://mlsanity.com/paper/2001.08177/) (2020)
- [Opponent Learning Awareness and Modelling in Multi-Objective Normal Form Games](https://mlsanity.com/paper/2011.07290/) (2020)
- [Scalable Optimization for Wind Farm Control using Coordination Graphs](https://mlsanity.com/paper/2101.07844/) (2021)
- [Risk Aware and Multi-Objective Decision Making with Distributional Monte Carlo Tree Search](https://mlsanity.com/paper/2102.00966/) (2021)
- [Approximation of Lorenz-Optimal Solutions in Multiobjective Markov Decision Processes](https://mlsanity.com/paper/1309.6856/) (2013)
- [Identification and Off-Policy Learning of Multiple Objectives Using Adaptive Clustering](https://mlsanity.com/paper/1705.06342/) (2017)
- [Fleetwide data-enabled reliability improvement of wind turbines](https://mlsanity.com/paper/1903.11518/) (2019)

## Built on by (most cited first, 30 of 256)

- [Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging](https://mlsanity.com/paper/2310.11564/) (2023)
- [Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards](https://mlsanity.com/paper/2306.04488/) (2023)
- [A Roadmap to Pluralistic Alignment](https://mlsanity.com/paper/2402.05070/) (2024)
- [Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment](https://mlsanity.com/paper/2402.10207/) (2024)
- [Secret Collusion among AI Agents: Multi-Agent Deception via Steganography](https://mlsanity.com/paper/2402.07510/) (2024)
- [Multi-Objective GFlowNets](https://mlsanity.com/paper/2210.12765/) (2022)
- [PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm](https://mlsanity.com/paper/2208.07914/) (2022)
- [Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)](https://mlsanity.com/paper/2112.15422/) (2021)
- [Value Augmented Sampling for Language Model Alignment and Personalization](https://mlsanity.com/paper/2405.06639/) (2024)
- [Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization](https://mlsanity.com/paper/2301.07784/) (2023)
- [Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning](https://mlsanity.com/paper/2407.15762/) (2024)
- [Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RL](https://mlsanity.com/paper/2305.00567/) (2023)
- [Multi-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Framework](https://mlsanity.com/paper/2311.12495/) (2023)
- [Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferences](https://mlsanity.com/paper/2312.09337/) (2023)
- [Pareto Conditioned Networks](https://mlsanity.com/paper/2204.05036/) (2022)
- [Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey](https://mlsanity.com/paper/2108.09003/) (2021)
- [Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weighting](https://mlsanity.com/paper/2509.11452/) (2025)
- [MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems](https://mlsanity.com/paper/2509.22047/) (2025)
- [Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer](https://mlsanity.com/paper/2206.11326/) (2022)
- [Deep Reinforcement Learning for Adaptive Mesh Refinement](https://mlsanity.com/paper/2209.12351/) (2022)
- [Pareto Set Learning for Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2501.06773/) (2025)
- [Foundations of Multivariate Distributional Reinforcement Learning](https://mlsanity.com/paper/2409.00328/) (2024)
- [MORL-Prompt: An Empirical Analysis of Multi-Objective Reinforcement Learning for Discrete Prompt Optimization](https://mlsanity.com/paper/2402.11711/) (2024)
- [Few for Many: Tchebycheff Set Scalarization for Many-Objective Optimization](https://mlsanity.com/paper/2405.19650/) (2024)
- [The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithm](https://mlsanity.com/paper/2406.07826/) (2024)
- [gym-DSSAT: a crop model turned into a Reinforcement Learning environment](https://mlsanity.com/paper/2207.03270/) (2022)
- [Stage-Wise Reward Shaping for Acrobatic Robots: A Constrained Multi-Objective Reinforcement Learning Approach](https://mlsanity.com/paper/2409.15755/) (2024)
- [Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models](https://mlsanity.com/paper/2505.10892/) (2025)
- [Vector Optimization with Stochastic Bandit Feedback](https://mlsanity.com/paper/2110.12311/) (2021)
- [Reward-free Alignment for Conflicting Objectives](https://mlsanity.com/paper/2602.02495/) (2026)

## Abstract (arXiv)

Real-world decision-making tasks are generally complex, requiring trade-offs between multiple, often conflicting, objectives. Despite this, the majority of research in reinforcement learning and decision-theoretic planning either assumes only a single objective, or that multiple objectives can be adequately handled via a simple linear combination. Such approaches may oversimplify the underlying problem and hence produce suboptimal results. This paper serves as a guide to the application of multi-objective methods to difficult problems, and is aimed at researchers who are already familiar with single-objective reinforcement learning and planning methods who wish to adopt a multi-objective perspective on their research, as well as practitioners who encounter multi-objective decision problems in practice. It identifies the factors that may influence the nature of the desired solution, and illustrates by example how these influence the design of multi-objective decision-making systems for complex problems.