# Expected Scalarised Returns Dominance: A New Solution Concept for Multi-Objective Decision Making

arXiv: https://arxiv.org/abs/2106.01048 · HTML: https://mlsanity.com/paper/2106.01048/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 49 references that are in the map.
- First posted on arXiv 2 Jun 2021.

## Builds on (most cited first, 10 of 10)

- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [A Survey of Multi-Objective Sequential Decision-Making](https://mlsanity.com/paper/1402.0590/) (2014)
- [A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation](https://mlsanity.com/paper/1908.08342/) (2019)
- [Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey](https://mlsanity.com/paper/1909.02964/) (2019)
- [Stochastically Dominant Distributional Reinforcement Learning](https://mlsanity.com/paper/1905.07318/) (2019)
- [A utility-based analysis of equilibria in multi-objective normal form games](https://mlsanity.com/paper/2001.08177/) (2020)
- [Risk Aware and Multi-Objective Decision Making with Distributional Monte Carlo Tree Search](https://mlsanity.com/paper/2102.00966/) (2021)
- [Combinatorial Multi-Objective Multi-Armed Bandit Problem](https://mlsanity.com/paper/1803.04039/) (2018)
- [Exploring the Impact of Tunable Agents in Sequential Social Dilemmas](https://mlsanity.com/paper/2101.11967/) (2021)

## Built on by (most cited first, 6 of 6)

- [Monte Carlo Tree Search Algorithms for Risk-Aware and Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2211.13032/) (2022)
- [What Lies beyond the Pareto Front? A Survey on Decision-Support Methods for Multi-Objective Optimization](https://mlsanity.com/paper/2311.11288/) (2023)
- [Multi-Objective Coordination Graphs for the Expected Scalarised Returns with Generative Flow Models](https://mlsanity.com/paper/2207.00368/) (2022)
- [Distributional Multi-Objective Decision Making](https://mlsanity.com/paper/2305.05560/) (2023)
- [FairDICE: Fairness-Driven Offline Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2506.08062/) (2025)
- [AETDICE: Unified Framework and Offline Optimization for Nonlinear Multi-Objective RL](https://mlsanity.com/paper/2606.31178/) (2026)

## Abstract (arXiv)

In many real-world scenarios, the utility of a user is derived from the single execution of a policy. In this case, to apply multi-objective reinforcement learning, the expected utility of the returns must be optimised. Various scenarios exist where a user's preferences over objectives (also known as the utility function) are unknown or difficult to specify. In such scenarios, a set of optimal policies must be learned. However, settings where the expected utility must be maximised have been largely overlooked by the multi-objective reinforcement learning community and, as a consequence, a set of optimal solutions has yet to be defined. In this paper we address this challenge by proposing first-order stochastic dominance as a criterion to build solution sets to maximise expected utility. We also propose a new dominance criterion, known as expected scalarised returns (ESR) dominance, that extends first-order stochastic dominance to allow a set of optimal policies to be learned in practice. We then define a new solution concept called the ESR set, which is a set of policies that are ESR dominant. Finally, we define a new multi-objective distributional tabular reinforcement learning (MOT-DRL) algorithm to learn the ESR set in a multi-objective multi-armed bandit setting.