# Utility-Based Reinforcement Learning: Unifying Single-objective and Multi-objective Reinforcement Learning

arXiv: https://arxiv.org/abs/2402.02665 · HTML: https://mlsanity.com/paper/2402.02665/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 10 of its 31 references that are in the map.
- First posted on arXiv 5 Feb 2024.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 10 of 10)

- [A Survey of Multi-Objective Sequential Decision-Making](https://mlsanity.com/paper/1402.0590/) (2014)
- [A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation](https://mlsanity.com/paper/1908.08342/) (2019)
- [Reward (Mis)design for Autonomous Driving](https://mlsanity.com/paper/2104.13906/) (2021)
- [Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)](https://mlsanity.com/paper/2112.15422/) (2021)
- [Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey](https://mlsanity.com/paper/1909.02964/) (2019)
- [Meta-Learning for Multi-objective Reinforcement Learning](https://mlsanity.com/paper/1811.03376/) (2018)
- [Improving Robustness via Risk Averse Distributional Reinforcement Learning](https://mlsanity.com/paper/2005.00585/) (2020)
- [Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer](https://mlsanity.com/paper/2206.11326/) (2022)
- [Stochastically Dominant Distributional Reinforcement Learning](https://mlsanity.com/paper/1905.07318/) (2019)
- [Monte Carlo Tree Search Algorithms for Risk-Aware and Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2211.13032/) (2022)

## Built on by (most cited first, 4 of 4)

- [Exploring Equity of Climate Policies using Multi-Agent Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2505.01115/) (2025)
- [GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfare](https://mlsanity.com/paper/2510.08872/) (2025)
- [Issues with Value-Based Multi-objective Reinforcement Learning: Value Function Interference and Overestimation Sensitivity](https://mlsanity.com/paper/2402.06266/) (2024)
- [It's a matter of timescale: non-linear utility in successor features and multi-objective planning and learning](https://mlsanity.com/paper/2608.25723/) (2026)

## Abstract (arXiv)

Research in multi-objective reinforcement learning (MORL) has introduced the utility-based paradigm, which makes use of both environmental rewards and a function that defines the utility derived by the user from those rewards. In this paper we extend this paradigm to the context of single-objective reinforcement learning (RL), and outline multiple potential benefits including the ability to perform multi-policy learning across tasks relating to uncertain objectives, risk-aware RL, discounting, and safe RL. We also examine the algorithmic implications of adopting a utility-based approach.