Limitations of Scalarisation in MORL: A Comparative Study in Discrete Environments
On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization
Abstract (arXiv)
Scalarisation functions are widely employed in MORL algorithms to enable intelligent decision-making. However, these functions often struggle to approximate the Pareto front accurately, rendering them unideal in complex, uncertain environments. This study examines selected Multi-Objective Reinforcement Learning (MORL) algorithms across MORL environments with discrete action and observation spaces. We aim to investigate further the limitations associated with scalarisation approaches for decision-making in multi-objective settings. Specifically, we use an outer-loop multi-policy methodology to assess the performance of a seminal single-policy MORL algorithm, MO Q-Learning implemented with linear scalarisation and Chebyshev scalarisation functions. In addition, we explore a pioneering inner-loop multi-policy algorithm, Pareto Q-Learning, which offers a more robust alternative. Our findings reveal that the performance of the scalarisation functions is highly dependent on the environment and the shape of the Pareto front. These functions often fail to retain the solutions uncovered during learning and favour finding solutions in certain regions of the solution space. Moreover, finding the appropriate weight configurations to sample the entire Pareto front is complex, limiting their applicability in uncertain settings. In contrast, inner-loop multi-policy algorithms may provide a more sustainable and generalizable approach and potentially facilitate intelligent decision-making in dynamic and uncertain environments.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 32 references that are in the map. First posted on arXiv 20 Nov 2025.
Builds on
12 of 32 references.
References · 12
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
- 2016Successor Features for Transfer in Reinforcement Learningcited by 340
- 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 256
- 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
- 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 177
- 2018Dynamic Weights in Multi-Objective Deep Reinforcement Learningcited by 100
- 2022PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithmcited by 58
- 2023Multi-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Frameworkcited by 31
- 2022Pareto Conditioned Networkscited by 25
- 2022Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfercited by 22
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learning
- 2024C-MORL: Multi-Objective Reinforcement Learning through Efficient Discovery of Pareto Front
- 2025Variance Reduced Policy Gradient Method for Multi-Objective Reinforcement Learning
- 2025On Generalization Across Environments In Multi-Objective Reinforcement Learning
- 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RL
- 2021On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuning
- 2025Pareto Set Learning for Multi-Objective Reinforcement Learning
- 2024Issues with Value-Based Multi-objective Reinforcement Learning: Value Function Interference and Overestimation Sensitivity