mlsanityData to 14 Oct 2026

Preference-based Multi-Objective Reinforcement Learning

arXiv 2507.14066 ↗First posted 18 Jul 2025cs.LGcited by 7

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

Multi-objective reinforcement learning (MORL) is a structured approach for optimizing tasks with multiple objectives. However, it often relies on pre-defined reward functions, which can be hard to design for balancing conflicting goals and may lead to oversimplification. Preferences can serve as more flexible and intuitive decision-making guidance, eliminating the need for complicated reward design. This paper introduces preference-based MORL (Pb-MORL), which formalizes the integration of preferences into the MORL framework. We theoretically prove that preferences can derive policies across the entire Pareto frontier. To guide policy optimization using preferences, our method constructs a multi-objective reward model that aligns with the given preferences. We further provide theoretical proof to show that optimizing this reward model is equivalent to training the Pareto optimal policy. Extensive experiments in benchmark multi-objective tasks, a multi-energy management task, and an autonomous driving task on a multi-line highway show that our method performs competitively, surpassing the oracle method, which uses the ground truth reward function. This highlights its potential for practical applications in complex real-world systems.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 24 of its 55 references that are in the map. First posted on arXiv 18 Jul 2025.

Builds on

24 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Human-in-the-Loop Policy Optimization for Preference-Based Multi-Objective Reinforcement Learning4 Jan

Other references · 23

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2016OpenAI Gymcited by 1,736
  6. 2018Exploration by Random Network Distillationcited by 993
  7. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  8. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  9. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 256
  10. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 177
  11. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  12. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 97
11 more
  1. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
  2. 2023Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritizationcited by 48
  3. 2021Self-Supervised Online Reward Shaping in Sparse-Reward Environmentscited by 37
  4. 2022Pareto Conditioned Networkscited by 25
  5. 2024Enhancing Autonomous Vehicle Training with Language Model Integration and Critical Scenario Generationcited by 19
  6. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 15
  7. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
  8. 2022E-MAPP: Efficient Multi-Agent Reinforcement Learning with Parallel Program Guidancecited by 6
  9. 2024S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learningcited by 4
  10. 2025CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queriescited by 3
  11. 2021Decentralized Multi-Agent Reinforcement Learning: An Off-Policy Methodcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learningcited by 3
  2. 2026PRISM: Parallel Reward Integration with Symmetry for MORLcited by 1
  3. 2026COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Spacecited by 1
  4. 2025BoreaRL: A Multi-Objective Reinforcement Learning Environment for Climate-Adaptive Boreal Forest Managementcited by 0
  5. 2026Adaptive Smooth Tchebycheff Attention for Multi-Objective Policy Optimizationcited by 0
  6. 2026On Generalizing Preference-based Reinforcement Learning to Multi-Objective Problemscited by 0
  7. 2026Multi-Agent Reinforcement Learning via Agent-Specific Preferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024C-MORL: Multi-Objective Reinforcement Learning through Efficient Discovery of Pareto Front
  2. 2026A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning
  3. 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RL
  4. 2024Utility-Based Reinforcement Learning: Unifying Single-objective and Multi-objective Reinforcement Learning
  5. 2024Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learning
  6. 2025Pareto Set Learning for Multi-Objective Reinforcement Learning
  7. 2025On Generalization Across Environments In Multi-Objective Reinforcement Learning
  8. 2026Constrained Multi-Objective Reinforcement Learning with Max-Min Criterion