mlsanityData to 7 Oct 2026

Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization

arXiv 2412.05469 ↗First posted 6 Dec 2024cs.LGcited by 20

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Multi-objective alignment from human feedback (MOAHF) in large language models (LLMs) is a challenging problem as human preferences are complex, multifaceted, and often conflicting. Recent works on MOAHF considered a-priori multi-objective optimization (MOO), where human preferences are known at training or inference time. In contrast, when human preferences are unknown or difficult to quantify, a natural approach is to cover the Pareto front by multiple diverse solutions. We propose an algorithm HaM for learning diverse LLM policies that maximizes their hypervolume. This is the first application of a-posteriori MOO to MOAHF. HaM is computationally and space efficient, and empirically superior across objectives such as harmlessness, helpfulness, humor, faithfulness, and hallucination, on various datasets.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 11 of its 35 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

11 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  2. 2024DeAL: Decoding-time Alignment for Large Language Models5 Feb
  3. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct

Other references · 8

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  8. 2023Instruction Tuning with GPT-4cited by 676

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 7.

20 papers.

Sort:

By area:

  1. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  2. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Modelscited by 22
  3. 2025From Selection to Generation: A Survey of LLM-based Active Learningcited by 18
  4. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Modelscited by 13
  5. 2025Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systemscited by 11
  6. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 10
  7. 2025MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learningcited by 9
  8. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Modelscited by 6
  9. 2025Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMscited by 4
  10. 2025DICE: A Framework for Dimensional and Contextual Evaluation of Language Modelscited by 3
  11. 2025SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimizationcited by 3
  12. 2026Focal Reward: Balanced Reinforcement Learning under Rubric-Based Rewardscited by 3
8 more
  1. 2026HIPO: Instruction Hierarchy via Constrained Reinforcement Learningcited by 2
  2. 2026Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimizationcited by 2
  3. 2026MOCHA: Multi-Objective Chebyshev Annealing for Agent Skill Optimizationcited by 1
  4. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  5. 2025TestDG: Test-time Domain Generalization for Continual Test-time Adaptationcited by 0
  6. 2025Multi-Objective Reward and Preference Optimization: Theory and Algorithmscited by 0
  7. 2026PAFO: Pareto Fairness Optimization for Personalized Reward Modelingcited by 0
  8. 2026MoMHa: Multi-Objective Optimization of LLM Harnesses over Accuracy, Safety, and Tokenscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026One Model for All: Multi-Objective Controllable Language Models
  2. 2025UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality
  3. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selection
  4. 2025Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
  5. 2025Pareto Multi-Objective Alignment for Language Models
  6. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards
  7. 2026Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
  8. 2025Robust Multi-Objective Preference Alignment with Online DPO