# Issues with Value-Based Multi-objective Reinforcement Learning: Value Function Interference and Overestimation Sensitivity

arXiv: https://arxiv.org/abs/2402.06266 · HTML: https://mlsanity.com/paper/2402.06266/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 9 Feb 2024.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 16 of 16)

- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [A Survey of Multi-Objective Sequential Decision-Making](https://mlsanity.com/paper/1402.0590/) (2014)
- [Deep Reinforcement Learning and the Deadly Triad](https://mlsanity.com/paper/1812.02648/) (2018)
- [Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.01929/) (2016)
- [Dynamic Weights in Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1809.07803/) (2018)
- [Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1610.02707/) (2016)
- [Learning Fair Policies in Multiobjective (Deep) Reinforcement Learning with Average and Discounted Rewards](https://mlsanity.com/paper/2008.07773/) (2020)
- [The Phenomenon of Policy Churn](https://mlsanity.com/paper/2206.00730/) (2022)
- [Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey](https://mlsanity.com/paper/2108.09003/) (2021)
- [Multi-condition multi-objective optimization using deep reinforcement learning](https://mlsanity.com/paper/2110.05945/) (2021)
- [Elastic Step DQN: A novel multi-step algorithm to alleviate overestimation in Deep QNetworks](https://mlsanity.com/paper/2210.03325/) (2022)
- [Utility-Based Reinforcement Learning: Unifying Single-objective and Multi-objective Reinforcement Learning](https://mlsanity.com/paper/2402.02665/) (2024)
- [Chance-Constrained Control with Lexicographic Deep Reinforcement Learning](https://mlsanity.com/paper/2010.09468/) (2020)
- [Relationship Explainable Multi-objective Reinforcement Learning with Semantic Explainability Generation](https://mlsanity.com/paper/1909.12268/) (2019)
- [An Empirical Investigation of Value-Based Multi-objective Reinforcement Learning for Stochastic Environments](https://mlsanity.com/paper/2401.03163/) (2024)

## Built on by (most cited first, 1 of 1)

- [On Generalization Across Environments In Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2503.00799/) (2025)

## Abstract (arXiv)

Multi-objective reinforcement learning (MORL) algorithms extend conventional reinforcement learning (RL) to the more general case of problems with multiple, conflicting objectives, represented by vector-valued rewards. Widely-used scalar RL methods such as Q-learning can be modified to handle multiple objectives by (1) learning vector-valued value functions, and (2) performing action selection using a scalarisation or ordering operator which reflects the user's preferences with respect to the different objectives. This paper investigates two previously unreported issues which can hinder the performance of value-based MORL algorithms when applied in conjunction with a non-linear utility function -- value function interference, and sensitivity to overestimation. We illustrate the nature of these phenomena on simple multi-objective MDPs using a tabular implementation of multiobjective Q-learning.