# Pessimistic Off-Policy Multi-Objective Optimization

arXiv: https://arxiv.org/abs/2310.18617 · HTML: https://mlsanity.com/paper/2310.18617/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 64 references that are in the map.
- First posted on arXiv 28 Oct 2023.

## Builds on (most cited first, 15 of 15)

- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Is Pessimism Provably Efficient for Offline RL?](https://mlsanity.com/paper/2012.15085/) (2020)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2006.05078/) (2020)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [A Flexible Framework for Multi-Objective Bayesian Optimization using Random Scalarizations](https://mlsanity.com/paper/1805.12168/) (2018)
- [Ensemble Sampling](https://mlsanity.com/paper/1705.07347/) (2017)
- [Random Hypervolume Scalarizations for Provable Multi-Objective Black Box Optimization](https://mlsanity.com/paper/2006.04655/) (2020)
- [Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RL](https://mlsanity.com/paper/2305.00567/) (2023)
- [Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequality](https://mlsanity.com/paper/2212.09900/) (2022)
- [Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs](https://mlsanity.com/paper/2106.00099/) (2021)
- [Multi-Objective Learning to Predict Pareto Fronts Using Hypervolume Maximization](https://mlsanity.com/paper/2102.04523/) (2021)
- [Multi-Task Off-Policy Learning from Bandit Feedback](https://mlsanity.com/paper/2212.04720/) (2022)
- [Multi-objective Optimization by Uncrowded Hypervolume Gradient Ascent](https://mlsanity.com/paper/2007.04846/) (2020)
- [IMO3: Interactive Multi-Objective Off-Policy Optimization](https://mlsanity.com/paper/2201.09798/) (2022)

## Built on by (most cited first, 2 of 2)

- [A General Framework for Off-Policy Learning with Partially-Observed Reward](https://mlsanity.com/paper/2506.14439/) (2025)
- [MoMHa: Multi-Objective Optimization of LLM Harnesses over Accuracy, Safety, and Tokens](https://mlsanity.com/paper/2609.30967/) (2026)

## Abstract (arXiv)

Multi-objective optimization is a type of decision making problems where multiple conflicting objectives are optimized. We study offline optimization of multi-objective policies from data collected by an existing policy. We propose a pessimistic estimator for the multi-objective policy values that can be easily plugged into existing formulas for hypervolume computation and optimized. The estimator is based on inverse propensity scores (IPS), and improves upon a naive IPS estimator in both theory and experiments. Our analysis is general, and applies beyond our IPS estimators and methods for optimizing them. The pessimistic estimator can be optimized by policy gradients and performs well in all of our experiments.