# Multi-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Framework

arXiv: https://arxiv.org/abs/2311.12495 · HTML: https://mlsanity.com/paper/2311.12495/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 77 references that are in the map.
- First posted on arXiv 21 Nov 2023.

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [A Survey of Multi-Objective Sequential Decision-Making](https://mlsanity.com/paper/1402.0590/) (2014)
- [pymoo: Multi-objective Optimization in Python](https://mlsanity.com/paper/2002.04504/) (2020)
- [A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation](https://mlsanity.com/paper/1908.08342/) (2019)
- [Dynamic Weights in Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1809.07803/) (2018)
- [Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1610.02707/) (2016)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Hyperparameters in Reinforcement Learning and How To Tune Them](https://mlsanity.com/paper/2306.01324/) (2023)
- [Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)](https://mlsanity.com/paper/2112.15422/) (2021)
- [Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization](https://mlsanity.com/paper/2301.07784/) (2023)
- [Hyperparameter Optimization for Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2310.16487/) (2023)
- [Distributional Multi-Objective Decision Making](https://mlsanity.com/paper/2305.05560/) (2023)

## Built on by (most cited first, 30 of 31)

- [Pareto Multi-Objective Alignment for Language Models](https://mlsanity.com/paper/2508.07768/) (2025)
- [Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyond](https://mlsanity.com/paper/2501.10945/) (2025)
- [AMOR: Adaptive Character Control through Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2505.23708/) (2025)
- [On Generalization Across Environments In Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2503.00799/) (2025)
- [MOMAland: A Set of Benchmarks for Multi-Objective Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2407.16312/) (2024)
- [Multi-objective Deep Learning: Taxonomy and Survey of the State of the Art](https://mlsanity.com/paper/2412.01566/) (2024)
- [C-MORL: Multi-Objective Reinforcement Learning through Efficient Discovery of Pareto Front](https://mlsanity.com/paper/2410.02236/) (2024)
- [Scalable Multi-Objective Reinforcement Learning with Fairness Guarantees using Lorenz Dominance](https://mlsanity.com/paper/2411.18195/) (2024)
- [Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization](https://mlsanity.com/paper/2602.07764/) (2026)
- [Exploring Equity of Climate Policies using Multi-Agent Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2505.01115/) (2025)
- [Reinforcement Learning for Multi-Objective Multi-Echelon Supply Chain Optimisation](https://mlsanity.com/paper/2507.19788/) (2025)
- [A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2604.24532/) (2026)
- [MOMA-AC: A preference-driven actor-critic framework for continuous multi-objective multi-agent reinforcement learning](https://mlsanity.com/paper/2511.18181/) (2025)
- [LacaDM: A Latent Causal Diffusion Model for Multiobjective Reinforcement Learning](https://mlsanity.com/paper/2512.19516/) (2025)
- [MIRACL: A Diverse Meta-Reinforcement Learning for Multi-Objective Multi-Echelon Combinatorial Supply Chain Optimisation](https://mlsanity.com/paper/2603.05760/) (2026)
- [ORACLE: A Multi-Objective Reinforcement Learning-Based Analog Circuit Design Optimizer with Large Language Models-Guided Exploration](https://mlsanity.com/paper/2608.04999/) (2026)
- [UCB-driven Utility Function Search for Multi-objective Reinforcement Learning](https://mlsanity.com/paper/2405.00410/) (2024)
- [Interpretability by Design for Efficient Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2506.04022/) (2025)
- [A Fairness-Oriented Multi-Objective Reinforcement Learning approach for Autonomous Intersection Management](https://mlsanity.com/paper/2507.09311/) (2025)
- [Multi-Objective Search: Algorithms, Applications, and Emerging Directions](https://mlsanity.com/paper/2510.25504/) (2025)
- [Limitations of Scalarisation in MORL: A Comparative Study in Discrete Environments](https://mlsanity.com/paper/2511.16476/) (2025)
- [Enabling Population-Based Architectures for Neural Combinatorial Optimization](https://mlsanity.com/paper/2601.08696/) (2026)
- [Optimizing Energy and Data Collection in UAV-aided IoT Networks using Attention-based Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2601.14092/) (2026)
- [Post Hoc Extraction of Pareto Fronts for Continuous Control](https://mlsanity.com/paper/2603.02628/) (2026)
- [Multi-objective Reinforcement Learning With Augmented States Requires Rewards After Deployment](https://mlsanity.com/paper/2604.15757/) (2026)
- [Adaptive Smooth Tchebycheff Attention for Multi-Objective Policy Optimization](https://mlsanity.com/paper/2605.12771/) (2026)
- [Disturbance-Aware Aerial Robotics for Ethical Wildlife Monitoring](https://mlsanity.com/paper/2606.08249/) (2026)
- [Objective-Behavior Alignment: Diagnostics for MORL Policy Selection](https://mlsanity.com/paper/2606.21321/) (2026)
- [Meta-Reinforcement Learning via Evolution for Multi-Objective Combinatorial Supply Chain Optimisation](https://mlsanity.com/paper/2606.22146/) (2026)
- [Sampling-Based Coordination-Informed Multi-Objective Multi-Robot Reinforcement Learning](https://mlsanity.com/paper/2606.30893/) (2026)

## Abstract (arXiv)

Multi-objective reinforcement learning (MORL) extends traditional RL by seeking policies making different compromises among conflicting objectives. The recent surge of interest in MORL has led to diverse studies and solving methods, often drawing from existing knowledge in multi-objective optimization based on decomposition (MOO/D). Yet, a clear categorization based on both RL and MOO/D is lacking in the existing literature. Consequently, MORL researchers face difficulties when trying to classify contributions within a broader context due to the absence of a standardized taxonomy. To tackle such an issue, this paper introduces multi-objective reinforcement learning based on decomposition (MORL/D), a novel methodology bridging the literature of RL and MOO. A comprehensive taxonomy for MORL/D is presented, providing a structured foundation for categorizing existing and potential MORL works. The introduced taxonomy is then used to scrutinize MORL research, enhancing clarity and conciseness through well-defined categorization. Moreover, a flexible framework derived from the taxonomy is introduced. This framework accommodates diverse instantiations using tools from both RL and MOO/D. Its versatility is demonstrated by implementing it in different configurations and assessing it on contrasting benchmark problems. Results indicate MORL/D instantiations achieve comparable performance to current state-of-the-art approaches on the studied problems. By presenting the taxonomy and framework, this paper offers a comprehensive perspective and a unified vocabulary for MORL. This not only facilitates the identification of algorithmic contributions but also lays the groundwork for novel research avenues in MORL.