# Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RL

arXiv: https://arxiv.org/abs/2305.00567 · HTML: https://mlsanity.com/paper/2305.00567/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 30 Apr 2023.

## Builds on (most cited first, 19 of 19)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [Distral: Robust Multitask Reinforcement Learning](https://mlsanity.com/paper/1707.04175/) (2017)
- [Multi-Game Decision Transformers](https://mlsanity.com/paper/2205.15241/) (2022)
- [Online Decision Transformer](https://mlsanity.com/paper/2202.05607/) (2022)
- [RvS: What is Essential for Offline RL via Supervised Learning?](https://mlsanity.com/paper/2112.10751/) (2021)
- [A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation](https://mlsanity.com/paper/1908.08342/) (2019)
- [Safe Policy Improvement with Baseline Bootstrapping](https://mlsanity.com/paper/1712.06924/) (2017)
- [Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1610.02707/) (2016)
- [A Distributional View on Multi-Objective Policy Optimization](https://mlsanity.com/paper/2005.07513/) (2020)
- [Meta-Learning for Multi-objective Reinforcement Learning](https://mlsanity.com/paper/1811.03376/) (2018)
- [Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs](https://mlsanity.com/paper/2106.00099/) (2021)
- [Relationship Explainable Multi-objective Optimization Via Vector Value Function Based Reinforcement Learning](https://mlsanity.com/paper/1910.01919/) (2019)

## Built on by (most cited first, 30 of 36)

- [Decoding-Time Language Model Alignment with Multiple Objectives](https://mlsanity.com/paper/2406.18853/) (2024)
- [Diffusion Models for Black-Box Optimization](https://mlsanity.com/paper/2306.07180/) (2023)
- [AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model](https://mlsanity.com/paper/2310.02054/) (2023)
- [Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback](https://mlsanity.com/paper/2402.02423/) (2024)
- [Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyond](https://mlsanity.com/paper/2501.10945/) (2025)
- [Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2407.17466/) (2024)
- [Generative Models in Decision Making: A Survey](https://mlsanity.com/paper/2502.17100/) (2025)
- [Offline Multi-Objective Optimization](https://mlsanity.com/paper/2406.03722/) (2024)
- [Semi-Offline Reinforcement Learning for Optimized Text Generation](https://mlsanity.com/paper/2306.09712/) (2023)
- [Policy-regularized Offline Multi-objective Reinforcement Learning](https://mlsanity.com/paper/2401.02244/) (2024)
- [Robust Multi-Objective Controlled Decoding of Large Language Models](https://mlsanity.com/paper/2503.08796/) (2025)
- [Decision Stacks: Flexible Reinforcement Learning via Modular Generative Models](https://mlsanity.com/paper/2306.06253/) (2023)
- [Partial Rankings of Optimizers](https://mlsanity.com/paper/2402.16565/) (2024)
- [OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment](https://mlsanity.com/paper/2509.24610/) (2025)
- [An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2409.09958/) (2024)
- [MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2408.15501/) (2024)
- [C-MORL: Multi-Objective Reinforcement Learning through Efficient Discovery of Pareto Front](https://mlsanity.com/paper/2410.02236/) (2024)
- [Agile Tradespace Exploration for Space Rendezvous Mission Design via Transformers](https://mlsanity.com/paper/2510.03544/) (2025)
- [Property-driven Protein Inverse Folding With Multi-Objective Preference Alignment](https://mlsanity.com/paper/2603.06748/) (2026)
- [Pessimistic Off-Policy Multi-Objective Optimization](https://mlsanity.com/paper/2310.18617/) (2023)
- [Reward Generation via Large Vision-Language Model in Offline Reinforcement Learning](https://mlsanity.com/paper/2504.08772/) (2025)
- [FairDICE: Fairness-Driven Offline Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2506.08062/) (2025)
- [GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learning](https://mlsanity.com/paper/2601.20753/) (2026)
- [Pareto Inverse Reinforcement Learning for Diverse Expert Policy Generation](https://mlsanity.com/paper/2408.12110/) (2024)
- [Preference-Conditioned Gradient Variations for Multi-Objective Quality-Diversity](https://mlsanity.com/paper/2411.12433/) (2024)
- [LacaDM: A Latent Causal Diffusion Model for Multiobjective Reinforcement Learning](https://mlsanity.com/paper/2512.19516/) (2025)
- [Interpretability by Design for Efficient Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2506.04022/) (2025)
- [Benchmarking Offline Multi-Objective Reinforcement Learning in Critical Care](https://mlsanity.com/paper/2512.08012/) (2025)
- [Post Hoc Extraction of Pareto Fronts for Continuous Control](https://mlsanity.com/paper/2603.02628/) (2026)
- [[Re] FairDICE: A Fair Tradeoff in Multi-objective Offline RL](https://mlsanity.com/paper/2603.03454/) (2026)

## Abstract (arXiv)

The goal of multi-objective reinforcement learning (MORL) is to learn policies that simultaneously optimize multiple competing objectives. In practice, an agent's preferences over the objectives may not be known apriori, and hence, we require policies that can generalize to arbitrary preferences at test time. In this work, we propose a new data-driven setup for offline MORL, where we wish to learn a preference-agnostic policy agent using only a finite dataset of offline demonstrations of other agents and their preferences. The key contributions of this work are two-fold. First, we introduce D4MORL, (D)atasets for MORL that are specifically designed for offline settings. It contains 1.8 million annotated demonstrations obtained by rolling out reference policies that optimize for randomly sampled preferences on 6 MuJoCo environments with 2-3 objectives each. Second, we propose Pareto-Efficient Decision Agents (PEDA), a family of offline MORL algorithms that builds and extends Decision Transformers via a novel preference-and-return-conditioned policy. Empirically, we show that PEDA closely approximates the behavioral policy on the D4MORL benchmark and provides an excellent approximation of the Pareto-front with appropriate conditioning, as measured by the hypervolume and sparsity metrics.