mlsanityData to 14 Oct 2026

A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning

arXiv 2604.24532 ↗First posted 27 Apr 2026cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

Many sequential decision-making tasks involve optimizing multiple conflicting objectives, requiring policies that adapt to different user preferences. In multi-objective reinforcement learning (MORL), one widely studied approach} addresses this by training a single policy network conditioned on preference-weighted rewards. In this paper, we explore a novel algorithmic perspective: leveraging reward-free reinforcement learning (RFRL) for MORL. While RFRL has historically been studied independently of MORL, it learns optimal policies for any possible reward function, making it a natural fit for MORL's challenge of handling unknown user preferences. We propose using the RFRL's training objective as an auxiliary task to enhance MORL, enabling more effective knowledge sharing beyond the multi-objective reward function given at training time. To this end, we adapt a state-of-the-art RFRL algorithm to the MORL setting and introduce a preference-guided exploration strategy that focuses learning on relevant parts of the environment. Through extensive experiments and ablation studies, we demonstrate that our approach significantly outperforms the state-of-the-art MORL methods across diverse MO-Gymnasium tasks, achieving superior performance and data efficiency. This work provides the first systematic adaptation of RFRL to MORL, demonstrating its potential as a scalable and empirically effective solution to multi-objective policy learning.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 48 of its 63 references that are in the map. First posted on arXiv 27 Apr 2026.

Builds on

48 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learning24 Jul
  2. 2024Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning5 May
  3. 2024Demonstration Guided Multi-Objective Reinforcement Learning5 Apr

Other references · 45

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  4. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  5. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  6. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 256
  7. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 177
  8. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  9. 2021Learning One Representation to Optimize All Rewardscited by 115
  10. 2018Universal Successor Features Approximatorscited by 103
  11. 2018Dynamic Weights in Multi-Objective Deep Reinforcement Learningcited by 100
  12. 2014Optimizing the CVaR via Samplingcited by 99
33 more
  1. 2024Foundation Policies with Hilbert Representationscited by 94
  2. 2020Fast active learning for pure exploration in reinforcement learningcited by 92
  3. 2020On Reward-Free Reinforcement Learning with Linear Function Approximationcited by 89
  4. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  5. 2016Multi-Objective Deep Reinforcement Learningcited by 83
  6. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  7. 2020Adaptive Reward-Free Explorationcited by 77
  8. 2016Deep Successor Reinforcement Learningcited by 72
  9. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
  10. 2022PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithmcited by 58
  11. 2020Provably Efficient Reward-Agnostic Navigation with Linear Value Iterationcited by 56
  12. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 56
  13. 2024Smooth Tchebycheff Scalarization for Multi-Objective Optimizationcited by 54
  14. 2020A Distributional View on Multi-Objective Policy Optimizationcited by 52
  15. 2016Deep Reinforcement Learning with Successor Features for Navigation across Similar Environmentscited by 50
  16. 2022Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processescited by 49
  17. 2023Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritizationcited by 48
  18. 2019Discovery of Useful Questions as Auxiliary Taskscited by 45
  19. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 42
  20. 2023Multi-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Frameworkcited by 31
  21. 2021Reward-Free Model-Based Reinforcement Learning with Linear Function Approximationcited by 29
  22. 2022Pareto Conditioned Networkscited by 25
  23. 2022Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfercited by 22
  24. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  25. 2021On Reward-Free RL with Kernel and Neural Function Approximations: Single-Agent MDP and Markov Gamecited by 20
  26. 2024Proto Successor Measure: Representing the Behavior Space of an RL Agentcited by 20
  27. 2021Gap-Dependent Unsupervised Exploration for Reinforcement Learningcited by 10
  28. 2024Learning Successor Features the Simple Waycited by 10
  29. 2022Q-Pensieve: Boosting Sample Efficiency of Multi-Objective RL Through Memory Sharing of Q-Snapshotscited by 9
  30. 2022Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learningcited by 6
  31. 2025Zero-Shot Reinforcement Learning Under Partial Observabilitycited by 6
  32. 2022What makes useful auxiliary tasks in reinforcement learning: investigating the effect of the target policycited by 5
  33. 2024SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learningcited by 1
  2. 2026On Generalizing Preference-based Reinforcement Learning to Multi-Objective Problemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Preference-based Multi-Objective Reinforcement Learning
  2. 2024C-MORL: Multi-Objective Reinforcement Learning through Efficient Discovery of Pareto Front
  3. 2025On Generalization Across Environments In Multi-Objective Reinforcement Learning
  4. 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RL
  5. 2024Utility-Based Reinforcement Learning: Unifying Single-objective and Multi-objective Reinforcement Learning
  6. 2026Multi-objective Reinforcement Learning With Augmented States Requires Rewards After Deployment
  7. 2026LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback
  8. 2021On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuning