mlsanityData to 7 Oct 2026

Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learning

arXiv 2407.17466 ↗First posted 24 Jul 2024cs.LG math.OC stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

This paper investigates multi-objective reinforcement learning (MORL), which focuses on learning Pareto optimal policies in the presence of multiple reward functions. Despite MORL's significant empirical success, there is still a lack of satisfactory understanding of various MORL optimization targets and efficient learning algorithms. Our work offers a systematic analysis of several optimization targets to assess their abilities to find all Pareto optimal policies and controllability over learned policies by the preferences for different objectives. We then identify Tchebycheff scalarization as a favorable scalarization method for MORL. Considering the non-smoothness of Tchebycheff scalarization, we reformulate its minimization problem into a new min-max-max optimization problem. Then, for the stochastic policy class, we propose efficient algorithms using this reformulation to learn Pareto optimal policies. We first propose an online UCB-based algorithm to achieve an varepsilon learning error with an tildeO(varepsilon-2) sample complexity for a single given preference. To further reduce the cost of environment exploration under different preferences, we propose a preference-free framework that first explores the environment without pre-defined preferences and then generates solutions for any number of preferences. We prove that it only requires an tildeO(varepsilon-2) exploration complexity in the exploration phase and demands no additional exploration afterward. Lastly, we analyze the smooth Tchebycheff scalarization, an extension of Tchebycheff scalarization, which is proved to be more advantageous in distinguishing the Pareto optimal policies from other weakly Pareto optimal policies based on entry values of preference vectors. Furthermore, we extend our algorithms and theoretical analysis to accommodate this optimization target.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 34 of its 95 references that are in the map. First posted on arXiv 24 Jul 2024.

Builds on

34 of 95 references.

References · 34

  1. 2017Hindsight Experience Replaycited by 1,096
  2. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  3. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  4. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  5. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 199
  6. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  7. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  8. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 166
  9. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  10. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 121
  11. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105
  12. 2020On Reward-Free Reinforcement Learning with Linear Function Approximationcited by 89
22 more
  1. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  2. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  3. 2023Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspectivecited by 51
  4. 2024Smooth Tchebycheff Scalarization for Multi-Objective Optimizationcited by 50
  5. 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RLcited by 36
  6. 2023Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidancecited by 35
  7. 2023Direction-oriented Multi-objective Learning: Simple and Provable Stochastic Algorithmscited by 33
  8. 2019Online Primal-Dual Mirror Descent under Stochastic Constraintscited by 31
  9. 2021A Simple Reward-free Approach to Constrained Reinforcement Learningcited by 28
  10. 2022On the Statistical Efficiency of Reward-Free Exploration in Non-Linear RLcited by 27
  11. 2023Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferencescited by 25
  12. 2021Provably Efficient Algorithms for Multi-Objective Competitive RLcited by 20
  13. 2018Multi-objective Contextual Bandit Problem with Similarity Informationcited by 19
  14. 2019Multi-Objective Generalized Linear Banditscited by 19
  15. 2022Anchor-Changing Regularized Natural Policy Gradient for Multi-Objective Reinforcement Learningcited by 15
  16. 2021Choosing the Best of Both Worlds: Diverse and Novel Recommendations through Multi-Objective Reinforcement Learningcited by 14
  17. 2023Improved Sample Complexity for Reward-free Reinforcement Learning under Low-rank MDPscited by 14
  18. 2022Near-Optimal Deployment Efficiency in Reward-Free Reinforcement Learning with Linear Function Approximationcited by 13
  19. 2020Accommodating Picky Customers: Regret Bound and Exploration Complexity for Multi-Objective Reinforcement Learningcited by 11
  20. 2022Provably Efficient Fictitious Play Policy Optimization for Zero-Sum Markov Games with Structured Transitionscited by 9
  21. 2023Optimal Horizon-Free Reward-Free Exploration for Linear Mixture MDPscited by 7
  22. 2020Gradient-Variation Bound for Online Convex Optimization with Constraintscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 9.

13 papers.

Sort:

By area:

  1. 2025Rethinking Diverse Human Preference Learning through Principal Component Analysiscited by 10
  2. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 10
  3. 2024How to Find the Exact Pareto Front for Multi-Objective MDPs?cited by 5
  4. 2026A Reward-Free Viewpoint on Multi-Objective Reinforcement Learningcited by 2
  5. 2026A Tale of Two Problems: Multi-Task Bilevel Learning Meets Equality Constrained Multi-Objective Optimizationcited by 1
  6. 2026Cost-Aware Multi-Objective Bandits: Theory and Application to Budgeted LLM Configuration Evaluationcited by 1
  7. 2025Enabling Pareto-Stationarity Exploration in Multi-Objective Reinforcement Learning: A Multi-Objective Weighted-Chebyshev Actor-Critic Approachcited by 0
  8. 2026Quality-Diversity Optimization as Multi-Objective Optimizationcited by 0
  9. 2026Pareto-Optimal Offline Reinforcement Learning via Smooth Tchebycheff Scalarizationcited by 0
  10. 2026Contextual Multi-Objective Optimization: Rethinking Objectives in Frontier AI Systemscited by 0
  11. 2026Adaptive Smooth Tchebycheff Attention for Multi-Objective Policy Optimizationcited by 0
  12. 2026Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparabilitycited by 0
1 more
  1. 2026Efficient Online Lexicographic Generalized Low-Rank Matrix Banditscited by 0