# Multi-Objective Coordination Graphs for the Expected Scalarised Returns with Generative Flow Models

arXiv: https://arxiv.org/abs/2207.00368 · HTML: https://mlsanity.com/paper/2207.00368/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 51 references that are in the map.
- First posted on arXiv 1 Jul 2022.

## Builds on (most cited first, 14 of 14)

- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design](https://mlsanity.com/paper/1902.00275/) (2019)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [A Survey of Multi-Objective Sequential Decision-Making](https://mlsanity.com/paper/1402.0590/) (2014)
- [Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)](https://mlsanity.com/paper/2112.15422/) (2021)
- [Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey](https://mlsanity.com/paper/1909.02964/) (2019)
- [Incremental Pruning: A Simple, Fast, Exact Method for Partially Observable Markov Decision Processes](https://mlsanity.com/paper/1302.1525/) (2013)
- [Exploring the Pareto front of multi-objective COVID-19 mitigation policies using reinforcement learning](https://mlsanity.com/paper/2204.05027/) (2022)
- [Expected Scalarised Returns Dominance: A New Solution Concept for Multi-Objective Decision Making](https://mlsanity.com/paper/2106.01048/) (2021)
- [Scalable Optimization for Wind Farm Control using Coordination Graphs](https://mlsanity.com/paper/2101.07844/) (2021)
- [Risk Aware and Multi-Objective Decision Making with Distributional Monte Carlo Tree Search](https://mlsanity.com/paper/2102.00966/) (2021)

## Built on by (most cited first, 3 of 3)

- [Monte Carlo Tree Search Algorithms for Risk-Aware and Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2211.13032/) (2022)
- [Finite-Time Frequentist Regret Bounds of Multi-Agent Thompson Sampling on Sparse Hypergraphs](https://mlsanity.com/paper/2312.15549/) (2023)
- [Deep Multi-Objective Reinforcement Learning for Utility-Based Infrastructural Maintenance Optimization](https://mlsanity.com/paper/2406.06184/) (2024)

## Abstract (arXiv)

Many real-world problems contain multiple objectives and agents, where a trade-off exists between objectives. Key to solving such problems is to exploit sparse dependency structures that exist between agents. For example, in wind farm control a trade-off exists between maximising power and minimising stress on the systems components. Dependencies between turbines arise due to the wake effect. We model such sparse dependencies between agents as a multi-objective coordination graph (MO-CoG). In multi-objective reinforcement learning a utility function is typically used to model a users preferences over objectives, which may be unknown a priori. In such settings a set of optimal policies must be computed. Which policies are optimal depends on which optimality criterion applies. If the utility function of a user is derived from multiple executions of a policy, the scalarised expected returns (SER) must be optimised. If the utility of a user is derived from a single execution of a policy, the expected scalarised returns (ESR) criterion must be optimised. For example, wind farms are subjected to constraints and regulations that must be adhered to at all times, therefore the ESR criterion must be optimised. For MO-CoGs, the state-of-the-art algorithms can only compute a set of optimal policies for the SER criterion, leaving the ESR criterion understudied. To compute a set of optimal polices under the ESR criterion, also known as the ESR set, distributions over the returns must be maintained. Therefore, to compute a set of optimal policies under the ESR criterion for MO-CoGs, we present a novel distributional multi-objective variable elimination (DMOVE) algorithm. We evaluate DMOVE in realistic wind farm simulations. Given the returns in real-world wind farm settings are continuous, we utilise a model known as real-NVP to learn the continuous return distributions to calculate the ESR set.