mlsanityData to 14 Oct 2026

MAP: Multi-Human-Value Alignment Palette

arXiv 2410.19198 ↗First posted 24 Oct 2024cs.AI cs.CY cs.ETcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Ensuring that generative AI systems align with human values is essential but challenging, especially when considering multiple human values and their potential trade-offs. Since human values can be personalized and dynamically change over time, the desirable levels of value alignment vary across different ethnic groups, industry sectors, and user cohorts. Within existing frameworks, it is hard to define human values and align AI systems accordingly across different directions simultaneously, such as harmlessness, helpfulness, and positiveness. To address this, we develop a novel, first-principle approach called Multi-Human-Value Alignment Palette (MAP), which navigates the alignment across multiple human values in a structured and reliable way. MAP formulates the alignment problem as an optimization task with user-defined constraints, which define human value targets. It can be efficiently solved via a primal-dual approach, which determines whether a user-defined alignment target is achievable and how to achieve it. We conduct a detailed theoretical analysis of MAP by quantifying the trade-offs between values, the sensitivity to constraints, the fundamental connection between multi-value alignment and sequential alignment, and proving that linear weighted rewards are sufficient for multi-value alignment. Extensive experiments demonstrate MAP's ability to align multiple values in a principled manner while delivering strong empirical performance across various tasks.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 31 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

17 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Contextual Moral Value Alignment Through Context-Based Aggregation19 Mar

Other references · 16

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  7. 2020Artificial Intelligence, Values and Alignmentcited by 448
  8. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  9. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  10. 2013A General Framework for Updating Belief Distributionscited by 175
  11. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  12. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 90
4 more
  1. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
  2. 2017Assigning a value to a power likelihood in a general Bayesian modelcited by 46
  3. 2018Model Selection Techniques -- An Overviewcited by 36
  4. 2023Language Model Decoding as Direct Metrics Optimizationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 11.

20 papers.

Sort:

By area:

  1. 2025Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughtscited by 19
  2. 2025Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probingcited by 14
  3. 2025An Outlook on the Opportunities and Challenges of Multi-Agent AI Systemscited by 8
  4. 2025OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignmentcited by 7
  5. 2025Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolationcited by 5
  6. 2025Towards Proactive Personalization through Profile Customization for Individual Users in Dialoguescited by 3
  7. 2025Aligning VLM Assistants with Personalized Situated Cognitioncited by 2
  8. 2026VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Modelscited by 2
  9. 2026Beyond Compromise: Pareto-Lenient Consensus for Efficient Multi-Preference LLM Alignmentcited by 2
  10. 2025PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolationcited by 1
  11. 2026Align When They Want, Complement When They Need! Human-Centered Ensembles for Adaptive Human-AI Collaborationcited by 1
  12. 2026Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansioncited by 1
8 more
  1. 2025DiverValue-Bench: A Benchmark and Fine-Tuning Framework for Aligning Large Language Models with Diverse Human Valuescited by 0
  2. 2026Adversarial Alignment: Ensuring Value Consistency in Large Language Models for Sensitive Domainscited by 0
  3. 2026Learning the Value Systems of Societies with Preference-based Multi-objective Reinforcement Learningcited by 0
  4. 2026Granite.Trust Policy Tools: Shareable, Actionable Policies for Generative AI Applicationscited by 0
  5. 2026Probing the Structure and Dynamics of LLM Value Expression through Value Conflictscited by 0
  6. 2026An Efficient and Modular Framework for Targeted Harm Mitigation in LLMScited by 0
  7. 2026From Alignment to Access Control: A Framework for GenAI Policy Enforcementcited by 0
  8. 2026AlphaPareto: Formulaic Alpha Discovery with LLM-Guided Multi-Objective Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models
  2. 2026Toward a Theory of Value in AI Alignment
  3. 2024Position: Towards Bidirectional Human-AI Alignment
  4. 2024ValueCompass: A Framework for Measuring Contextual Value Alignment Between Human and LLMs
  5. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models
  6. 2024What are human values, and how do we align AI to them?
  7. 2024A Roadmap to Pluralistic Alignment
  8. 2025Diverse Human Value Alignment for Large Language Models via Ethical Reasoning