mlsanityData to 14 Oct 2026

Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models

arXiv 2505.10892 ↗First posted 16 May 2025cs.LGcited by 13

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Post-training LLMs with RLHF and preference optimization methods (e.g., DPO, IPO) has greatly improved alignment, yet these approaches assume a single objective. In reality, humans express multiple, often conflicting objectives, such as helpfulness and harmlessness, with no natural scalarization. We study the multi-objective preference alignment problem, where a policy must balance several objectives simultaneously. We propose Multi-Objective Preference Optimization (MOPO), a constrained KL-regularized framework that maximizes a primary objective while enforcing lower bounds on secondary objectives via tunable safety thresholds. MOPO operates directly on pairwise preferences without point-wise rewards, and admits simple closed-form iterative updates. Empirically, MOPO recovers Pareto-optimal policies on synthetic benchmarks and, when fine-tuned on human-preference data, yields multi-billion parameter models that achieve higher rewards and Pareto-dominate baselines, with stable and robust optimization dynamics.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 73 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

56 of 73 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model6 May
  2. 2025Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment20 Feb
  3. 2024Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization6 Dec
  4. 2024L3Ms -- Lagrange Large Language Models28 Oct
  5. 2024Decoding-Time Language Model Alignment with Multiple Objectives27 Jun
  6. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  7. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  8. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  9. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs3 Feb
  10. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  12. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
3 more
  1. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  2. 2023Aligning Language Models with Offline Learning from Human Feedback23 Aug
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 41

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  10. 2020Learning to summarize from human feedbackcited by 2,588
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
29 more
  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  2. 2025Gemma 3 Technical Reportcited by 1,824
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  4. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  5. 2023Instruction Tuning with GPT-4cited by 686
  6. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  7. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  8. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  9. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  10. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  11. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 262
  12. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 256
  13. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  14. 2024Iterative Reasoning Preference Optimizationcited by 246
  15. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  16. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  17. 2023Nash Learning from Human Feedbackcited by 224
  18. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 90
  19. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  20. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 41
  21. 2019Constrained Deep Networks: Lagrangian Optimization via Log-Barrier Extensionscited by 30
  22. 2021Investigating the Impact of Multi-LiDAR Placement on Object Detection for Autonomous Drivingcited by 22
  23. 2023A Stochastic-Gradient-based Interior-Point Algorithm for Solving Smooth Bound-Constrained Optimization Problemscited by 8
  24. 2024Online Bandit Learning with Offline Preference Data for Improved RLHFcited by 6
  25. 2025Best Policy Learning from Trajectory Preference Feedbackcited by 5
  26. 2019A Convolutional Neural Network Approach Towards Self-Driving Carscited by 4
  27. 2023ACPO: A Policy Optimization Algorithm for Average MDPs with Constraintscited by 4
  28. 2024C-MORL: Multi-Objective Reinforcement Learning through Efficient Discovery of Pareto Frontcited by 4
  29. 2024e-COP : Episodic Constrained Optimization of Policiescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 10.

13 papers.

Sort:

By area:

  1. 2025PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimizationcited by 3
  2. 2025MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Modelscited by 3
  3. 2026GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimizationcited by 3
  4. 2026Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPOcited by 1
  5. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  6. 2026Freeform Preference Learning for Robotic Manipulationcited by 1
  7. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selectioncited by 0
  8. 2026Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuningcited by 0
  9. 2026Provable Last-Iterate Convergence for Multi-Objective Safe LLM Alignment via Optimistic Primal-Dualcited by 0
  10. 2026MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignmentcited by 0
  11. 2026Multi-Objective Exploration and Preference Optimization via Mutual Informationcited by 0
  12. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0
1 more
  1. 2026PoEM: Predicting RL Outcomes from Existing Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Robust Multi-Objective Preference Alignment with Online DPO
  2. 2025UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality
  3. 2026One Model for All: Multi-Objective Controllable Language Models
  4. 2025MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment
  5. 2026MixDPO: Modeling Preference Strength for Pluralistic Alignment
  6. 2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling
  7. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models
  8. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models