# Adaptive Alignment: Dynamic Preference Adjustments via Multi-Objective Reinforcement Learning for Pluralistic AI

arXiv: https://arxiv.org/abs/2410.23630 · HTML: https://mlsanity.com/paper/2410.23630/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Builds on (most cited first, 17 of 17)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards](https://mlsanity.com/paper/2306.04488/) (2023)
- [Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment](https://mlsanity.com/paper/2402.10207/) (2024)
- [Interactive Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1701.06049/) (2017)
- [Reward-rational (implicit) choice: A unifying formalism for reward learning](https://mlsanity.com/paper/2002.04833/) (2020)
- [Dynamic Weights in Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1809.07803/) (2018)
- [Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties](https://mlsanity.com/paper/2309.00779/) (2023)
- [AI Alignment and Social Choice: Fundamental Limitations and Policy Implications](https://mlsanity.com/paper/2310.16048/) (2023)
- [Towards Scalable Automated Alignment of LLMs: A Survey](https://mlsanity.com/paper/2406.01252/) (2024)
- [Active Learning in Robotics: A Review of Control Principles](https://mlsanity.com/paper/2106.13697/) (2021)
- [MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning](https://mlsanity.com/paper/2201.00012/) (2021)
- [Algorithmic Pluralism: A Structural Approach To Equal Opportunity](https://mlsanity.com/paper/2305.08157/) (2023)

## Built on by (most cited first, 6 of 6)

- [Virtual Agent Economies](https://mlsanity.com/paper/2509.10147/) (2025)
- [Direct Alignment with Heterogeneous Preferences](https://mlsanity.com/paper/2502.16320/) (2025)
- [LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spaces](https://mlsanity.com/paper/2503.01894/) (2025)
- [Overton Pluralistic Reinforcement Learning for Large Language Models](https://mlsanity.com/paper/2602.20759/) (2026)
- [From Sycophantic Consensus to Pluralistic Repair: Why AI Alignment Must Surface Disagreement](https://mlsanity.com/paper/2605.14912/) (2026)
- [Meta-ethics and AI: exploring the novel meta-ethical questions in the era of AI](https://mlsanity.com/paper/2609.01685/) (2026)

## Abstract (arXiv)

Emerging research in Pluralistic Artificial Intelligence (AI) alignment seeks to address how intelligent systems can be designed and deployed in accordance with diverse human needs and values. We contribute to this pursuit with a dynamic approach for aligning AI with diverse and shifting user preferences through Multi Objective Reinforcement Learning (MORL), via post-learning policy selection adjustment. In this paper, we introduce the proposed framework for this approach, outline its anticipated advantages and assumptions, and discuss technical details about the implementation. We also examine the broader implications of adopting a retroactive alignment approach through the sociotechnical systems perspective.