# Pareto Set Learning for Multi-Objective Reinforcement Learning

arXiv: https://arxiv.org/abs/2501.06773 · HTML: https://mlsanity.com/paper/2501.06773/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 114 of its 306 references that are in the map.
- First posted on arXiv 12 Jan 2025.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 30 of 114)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Benchmarking Graph Neural Networks](https://mlsanity.com/paper/2003.00982/) (2020)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Attention, Learn to Solve Routing Problems!](https://mlsanity.com/paper/1803.08475/) (2018)
- [Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizon](https://mlsanity.com/paper/1811.06128/) (2018)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Neural Combinatorial Optimization with Reinforcement Learning](https://mlsanity.com/paper/1611.09940/) (2016)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Learning Combinatorial Optimization Algorithms over Graphs](https://mlsanity.com/paper/1704.01665/) (2017)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Reinforcement Learning for Solving the Vehicle Routing Problem](https://mlsanity.com/paper/1802.04240/) (2018)
- [Exact Combinatorial Optimization with Graph Convolutional Neural Networks](https://mlsanity.com/paper/1906.01629/) (2019)
- [Size-Independent Sample Complexity of Neural Networks](https://mlsanity.com/paper/1712.06541/) (2017)

## Built on by (most cited first, 20 of 20)

- [PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model](https://mlsanity.com/paper/2505.06274/) (2025)
- [Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyond](https://mlsanity.com/paper/2501.10945/) (2025)
- [Ctrl-DNA: Controllable Cell-Type-Specific Regulatory DNA Design via Constrained RL](https://mlsanity.com/paper/2505.20578/) (2025)
- [Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization](https://mlsanity.com/paper/2602.07764/) (2026)
- [UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignment](https://mlsanity.com/paper/2602.09538/) (2026)
- [GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learning](https://mlsanity.com/paper/2601.20753/) (2026)
- [MO-Playground: Massively Parallelized Multi-Objective Reinforcement Learning for Robotics](https://mlsanity.com/paper/2603.09237/) (2026)
- [A Framework for Controllable Multi-objective Learning with Annealed Stein Variational Hypernetworks](https://mlsanity.com/paper/2506.06715/) (2025)
- [LacaDM: A Latent Causal Diffusion Model for Multiobjective Reinforcement Learning](https://mlsanity.com/paper/2512.19516/) (2025)
- [PRISM: Parallel Reward Integration with Symmetry for MORL](https://mlsanity.com/paper/2602.18277/) (2026)
- [A Single Deep Preference-Conditioned Policy for Learning Pareto Coverage Sets](https://mlsanity.com/paper/2605.08946/) (2026)
- [LLM-PDESR: Robust PDE Discovery via Subdomain Weighted Residuals and LLM-Guided Symbolic Hypothesis Generation](https://mlsanity.com/paper/2607.10546/) (2026)
- [ORACLE: A Multi-Objective Reinforcement Learning-Based Analog Circuit Design Optimizer with Large Language Models-Guided Exploration](https://mlsanity.com/paper/2608.04999/) (2026)
- [SLA-MORL: SLA-Aware Multi-Objective Reinforcement Learning for HPC Resource Optimization](https://mlsanity.com/paper/2508.03509/) (2025)
- [Intelligence as Trajectory-Dominant Pareto Optimization](https://mlsanity.com/paper/2602.13230/) (2026)
- [Post Hoc Extraction of Pareto Fronts for Continuous Control](https://mlsanity.com/paper/2603.02628/) (2026)
- [FlexiGrad: Adaptive Gradient Modulation for Hierarchical Fine-Grained Classification](https://mlsanity.com/paper/2607.17563/) (2026)
- [MOCHA: Multi-Objective Co-Design using Hypernetwork Architectures](https://mlsanity.com/paper/2609.30570/) (2026)
- [MoFlow: Multi-Objective Agentic Workflow Generation](https://mlsanity.com/paper/2609.38294/) (2026)
- [PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots](https://mlsanity.com/paper/2610.01260/) (2026)

## Abstract (arXiv)

Multi-objective decision-making problems have emerged in numerous real-world scenarios, such as video games, navigation and robotics. Considering the clear advantages of Reinforcement Learning (RL) in optimizing decision-making processes, researchers have delved into the development of Multi-Objective RL (MORL) methods for solving multi-objective decision problems. However, previous methods either cannot obtain the entire Pareto front, or employ only a single policy network for all the preferences over multiple objectives, which may not produce personalized solutions for each preference. To address these limitations, we propose a novel decomposition-based framework for MORL, Pareto Set Learning for MORL (PSL-MORL), that harnesses the generation capability of hypernetwork to produce the parameters of the policy network for each decomposition weight, generating relatively distinct policies for various scalarized subproblems with high efficiency. PSL-MORL is a general framework, which is compatible for any RL algorithm. The theoretical result guarantees the superiority of the model capacity of PSL-MORL and the optimality of the obtained policy network. Through extensive experiments on diverse benchmarks, we demonstrate the effectiveness of PSL-MORL in achieving dense coverage of the Pareto front, significantly outperforming state-of-the-art MORL methods in the hypervolume and sparsity indicators.