# Human-in-the-Loop Policy Optimization for Preference-Based Multi-Objective Reinforcement Learning

arXiv: https://arxiv.org/abs/2401.02160 · HTML: https://mlsanity.com/paper/2401.02160/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 39 of its 107 references that are in the map.
- First posted on arXiv 4 Jan 2024.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 30 of 39)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [A Survey of Multi-Objective Sequential Decision-Making](https://mlsanity.com/paper/1402.0590/) (2014)
- [A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation](https://mlsanity.com/paper/1908.08342/) (2019)
- [Deep Reinforcement Learning for Multi-objective Optimization](https://mlsanity.com/paper/1906.02386/) (2019)
- [Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1610.02707/) (2016)
- [A Distributional View on Multi-Objective Policy Optimization](https://mlsanity.com/paper/2005.07513/) (2020)
- [Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey](https://mlsanity.com/paper/1909.02964/) (2019)
- [Meta-Learning for Multi-objective Reinforcement Learning](https://mlsanity.com/paper/1811.03376/) (2018)
- [MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning](https://mlsanity.com/paper/2201.00012/) (2021)
- [Ordered Preference Elicitation Strategies for Supporting Multi-Objective Decision Making](https://mlsanity.com/paper/1802.07606/) (2018)
- [InstOptima: Evolutionary Multi-objective Instruction Optimization via Large Language Model-based Instruction Operators](https://mlsanity.com/paper/2310.17630/) (2023)
- [BiLO-CPDP: Bi-Level Programming for Automated Model Discovery in Cross-Project Defect Prediction](https://mlsanity.com/paper/2008.13489/) (2020)
- [Two-Archive Evolutionary Algorithm for Constrained Multi-Objective Optimization](https://mlsanity.com/paper/1711.07907/) (2017)
- [DeepSQLi: Deep Semantic Learning for Testing SQL Injection](https://mlsanity.com/paper/2005.11728/) (2020)
- [Dynamic Multi-Objectives Optimization with a Changing Number of Objectives](https://mlsanity.com/paper/1608.06514/) (2016)
- [Constrained Bayesian Optimization Under Partial Observations: Balanced Improvements and Provable Convergence](https://mlsanity.com/paper/2312.03212/) (2023)
- [Does Preference Always Help? A Holistic Study on Preference-Based Evolutionary Multi-Objective Optimisation Using Reference Points](https://mlsanity.com/paper/1909.13567/) (2019)
- [A Data-Driven Evolutionary Transfer Optimization for Expensive Problems in Dynamic Environments](https://mlsanity.com/paper/2211.02879/) (2022)
- [Interactive Decomposition Multi-Objective Optimization via Progressively Learned Value Functions](https://mlsanity.com/paper/1801.00609/) (2018)
- [An Improved Two-Archive Evolutionary Algorithm for Constrained Multi-Objective Optimization](https://mlsanity.com/paper/2103.06382/) (2021)
- [Evolutionary Many-Objective Optimization Based on Adversarial Decomposition](https://mlsanity.com/paper/1704.02340/) (2017)
- [Which Surrogate Works for Empirical Performance Modelling? A Case Study with Differential Evolution](https://mlsanity.com/paper/1901.11120/) (2019)
- [Visualisation of Pareto Front Approximation: A Short Survey and Empirical Comparisons](https://mlsanity.com/paper/1903.01768/) (2019)
- [Bayesian Network Based Label Correlation Analysis For Multi-label Classifier Chain](https://mlsanity.com/paper/1908.02172/) (2019)
- [On the Combined Impact of Population Size and Sub-problem Selection in MOEA/D](https://mlsanity.com/paper/2004.06961/) (2020)

## Built on by (most cited first, 4 of 4)

- [Preference-based Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2507.14066/) (2025)
- [Humans Coexist, So Must Embodied Artificial Agents](https://mlsanity.com/paper/2502.04809/) (2025)
- [ECO: Energy-Constrained Optimization with Reinforcement Learning for Humanoid Walking](https://mlsanity.com/paper/2602.06445/) (2026)
- [TANGO: RNA Topology and Geometry Co-Design](https://mlsanity.com/paper/2610.20277/) (2026)

## Abstract (arXiv)

Multi-objective reinforcement learning (MORL) aims to find a set of high-performing and diverse policies that address trade-offs between multiple conflicting objectives. However, in practice, decision makers (DMs) often deploy only one or a limited number of trade-off policies. Providing too many diversified trade-off policies to the DM not only significantly increases their workload but also introduces noise in multi-criterion decision-making. With this in mind, we propose a human-in-the-loop policy optimization framework for preference-based MORL that interactively identifies policies of interest. Our method proactively learns the DM's implicit preference information without requiring any a priori knowledge, which is often unavailable in real-world black-box decision scenarios. The learned preference information is used to progressively guide policy optimization towards policies of interest. We evaluate our approach against three conventional MORL algorithms that do not consider preference information and four state-of-the-art preference-based MORL algorithms on two MORL environments for robot control and smart grid management. Experimental results fully demonstrate the effectiveness of our proposed method in comparison to the other peer algorithms.