# Deterministic Pareto-Optimal Policy Synthesis for Multi-Objective Reinforcement Learning

arXiv: https://arxiv.org/abs/2606.26397 · HTML: https://mlsanity.com/paper/2606.26397/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 8 of its 42 references that are in the map.
- First posted on arXiv 24 Jun 2026.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 8 of 8)

- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [A Survey of Multi-Objective Sequential Decision-Making](https://mlsanity.com/paper/1402.0590/) (2014)
- [Optimization of Molecules via Deep Reinforcement Learning](https://mlsanity.com/paper/1810.08678/) (2018)
- [A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation](https://mlsanity.com/paper/1908.08342/) (2019)
- [Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning](https://mlsanity.com/paper/1903.08542/) (2019)
- [Automated Circuit Sizing with Multi-objective Optimization based on Differential Evolution and Bayesian Inference](https://mlsanity.com/paper/2206.02391/) (2022)
- [A Counterexample and a Corrective to the Vector Extension of the Bellman Equations of a Markov Decision Process](https://mlsanity.com/paper/2306.16937/) (2023)
- [Multi-Objective Learning Model Predictive Control](https://mlsanity.com/paper/2405.11698/) (2024)

## Abstract (arXiv)

Real-world decision-making often requires balancing multiple conflicting objectives, a challenge that standard Reinforcement Learning (RL) frequently addresses by aggregating rewards into a single scalar signal. While effective for simple tasks, this approach often fails to capture the full spectrum of optimal trade-offs, known as the Pareto frontier. In this paper, we introduce a novel preference-conditioned Bellman operator, motivated from the Chebyshev scalarization, designed to compute deterministic Pareto-optimal policies for Multi-Objective Markov Decision Processes (MOMDPs). We prove that this operator satisfies an enveloping property, where the estimated value functions upper-bound the true Pareto frontier, and demonstrate that it monotonically converges to a coverage set of this frontier. Furthermore, we also show how to extract deterministic policies from these converged Q-estimates. This ensures the agent can recover a policy for any given preference, capturing the entire Pareto-optimal frontier while guaranteeing each synthesized policy remains approximately Pareto-optimal. Experimental results validate that our algorithm successfully recovers complex trade-offs, providing a solution for deterministic Pareto-optimal policy synthesis.