# A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation

arXiv: https://arxiv.org/abs/1908.08342 · HTML: https://mlsanity.com/paper/1908.08342/

## Key facts

- Cited by 166 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 21 Aug 2019.

## Built on by (most cited first, 30 of 166)

- [Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging](https://mlsanity.com/paper/2310.11564/) (2023)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards](https://mlsanity.com/paper/2306.04488/) (2023)
- [A Roadmap to Pluralistic Alignment](https://mlsanity.com/paper/2402.05070/) (2024)
- [Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods](https://mlsanity.com/paper/2404.00282/) (2024)
- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [Multi-Objective GFlowNets](https://mlsanity.com/paper/2210.12765/) (2022)
- [LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts](https://mlsanity.com/paper/2501.00274/) (2024)
- [Panacea: Pareto Alignment via Preference Adaptation for LLMs](https://mlsanity.com/paper/2402.02030/) (2024)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm](https://mlsanity.com/paper/2208.07914/) (2022)
- [Pareto Set Learning for Expensive Multi-Objective Optimization](https://mlsanity.com/paper/2210.08495/) (2022)
- [Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization](https://mlsanity.com/paper/2301.07784/) (2023)
- [Scalable Pareto Front Approximation for Deep Multi-Objective Learning](https://mlsanity.com/paper/2103.13392/) (2021)
- [Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RL](https://mlsanity.com/paper/2305.00567/) (2023)
- [Multi-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Framework](https://mlsanity.com/paper/2311.12495/) (2023)
- [Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferences](https://mlsanity.com/paper/2312.09337/) (2023)
- [MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning](https://mlsanity.com/paper/2201.00012/) (2021)
- [Redeeming Intrinsic Rewards via Constrained Optimization](https://mlsanity.com/paper/2211.07627/) (2022)
- [Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs](https://mlsanity.com/paper/2106.00099/) (2021)
- [Pareto Conditioned Networks](https://mlsanity.com/paper/2204.05036/) (2022)
- [Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer](https://mlsanity.com/paper/2206.11326/) (2022)
- [Welfare and Fairness in Multi-objective Reinforcement Learning](https://mlsanity.com/paper/2212.01382/) (2022)
- [Pareto Multi-Objective Alignment for Language Models](https://mlsanity.com/paper/2508.07768/) (2025)
- [Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems](https://mlsanity.com/paper/2210.15432/) (2022)
- [Pareto Set Learning for Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2501.06773/) (2025)
- [GARDO: Reinforcing Diffusion Models without Reward Hacking](https://mlsanity.com/paper/2512.24138/) (2025)
- [MORL-Prompt: An Empirical Analysis of Multi-Objective Reinforcement Learning for Discrete Prompt Optimization](https://mlsanity.com/paper/2402.11711/) (2024)
- [Anchor-Changing Regularized Natural Policy Gradient for Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2206.05357/) (2022)
- [Socially Fair Reinforcement Learning](https://mlsanity.com/paper/2208.12584/) (2022)

## Abstract (arXiv)

We introduce a new algorithm for multi-objective reinforcement learning (MORL) with linear preferences, with the goal of enabling few-shot adaptation to new tasks. In MORL, the aim is to learn policies over multiple competing objectives whose relative importance (preferences) is unknown to the agent. While this alleviates dependence on scalar reward design, the expected return of a policy can change significantly with varying preferences, making it challenging to learn a single model to produce optimal policies under different preference conditions. We propose a generalized version of the Bellman equation to learn a single parametric representation for optimal policies over the space of all possible preferences. After an initial learning phase, our agent can execute the optimal policy under any given preference, or automatically infer an underlying preference with very few samples. Experiments across four different domains demonstrate the effectiveness of our approach.