mlsanityData to 7 Oct 2026

Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization

arXiv 2310.03708 ↗First posted 5 Oct 2023cs.LG cs.AIcited by 121

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

A single language model, even when aligned with labelers through reinforcement learning from human feedback (RLHF), may not suit all human preferences. Recent approaches therefore prefer customization, gathering multi-dimensional feedback, and creating distinct reward models for each dimension. Different language models are then optimized for various preferences using multi-objective RLHF (MORLHF) with varying reward weights. However, RL fine-tuning is unstable and resource-heavy, especially with diverse and usually conflicting objectives. In this paper, we present Multi-Objective Direct Preference Optimization (MODPO), an RL-free extension of Direct Preference Optimization (DPO) for multiple alignment objectives. Essentially, MODPO folds language modeling directly into reward modeling, training language models as implicit collective reward models that combine all objectives with specific weights. MODPO theoretically yields the same optimal solutions as MORLHF but is practically more stable and efficient. Empirical results in safety alignment and long-form question answering show that MODPO matches or outperforms existing methods, producing a Pareto front of language models catering to diverse preferences with three times less computational resources compared to MORLHF. Code is available at https://github.com/ZHZisZZ/modpo.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 32 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

21 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  2. 2023Preference Ranking Optimization for Human Alignment30 Jun
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
6 more
  1. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  4. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  5. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  6. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 87

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 61; 2026: 39.

121 papers.

Sort:

By area:

  1. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  2. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  3. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  4. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  5. 2024Language Models Resist Alignment: Evidence From Data Compressioncited by 50
  6. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 42
  7. 2024Online DPO: Online Direct Preference Optimization with Fast-Slow Chasingcited by 38
  8. 2025A Survey of Direct Preference Optimizationcited by 35
  9. 2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedbackcited by 31
  10. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  11. 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Modelscited by 27
  12. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
38 more
  1. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 27
  2. 2025ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoningcited by 24
  3. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  4. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  5. 2025Robust Multi-Objective Preference Alignment with Online DPOcited by 21
  6. 2024MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximationcited by 20
  7. 2024Multi-Objective Alignment of Large Language Models Through Hypervolume Maximizationcited by 20
  8. 2025Sotopia-RL: Reward Design for Social Intelligencecited by 20
  9. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  10. 2025Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundarycited by 17
  11. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 15
  12. 2025Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignmentcited by 15
  13. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  14. 2025PrefPalette: Personalized Preference Modeling with Latent Attributescited by 15
  15. 2024Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learningcited by 13
  16. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Modelscited by 13
  17. 2025Learning Safety Constraints for Large Language Modelscited by 13
  18. 2025CoSteer: Collaborative Decoding-Time Personalization via Local Delta Steeringcited by 12
  19. 2024Constraint Back-translation Improves Complex Instruction Following of Large Language Modelscited by 11
  20. 2025Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systemscited by 11
  21. 2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgmentcited by 11
  22. 2025Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimizationcited by 11
  23. 2026VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactionscited by 11
  24. 2025MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignmentcited by 10
  25. 2025CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectivescited by 10
  26. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  27. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 10
  28. 2025Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Modelscited by 10
  29. 2026Reward-free Alignment for Conflicting Objectivescited by 10
  30. 2025Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHFcited by 9
  31. 2025Societal Alignment Frameworks Can Improve LLM Alignmentcited by 8
  32. 2024Pareto-Optimal Energy Alignment for Designing Nature-Like Antibodiescited by 7
  33. 2025The Rise of Darkness: Safety-Utility Trade-Offs in Role-Playing Dialogue Agentscited by 7
  34. 2025Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codescited by 7
  35. 2025OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignmentcited by 7
  36. 2024Aligning Transformers with Continuous Feedback via Energy Rank Alignmentcited by 6
  37. 2025BalancedDPO: Adaptive Multi-Metric Alignmentcited by 6
  38. 2025Towards Understanding and Improving Refusal in Compressed Models via Mechanistic Interpretabilitycited by 6

50 most cited of 121 All 121 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  2. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  3. 2024New Desiderata for Direct Preference Optimization
  4. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
  5. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
  6. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model
  7. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models
  8. 2024Towards Efficient Exact Optimization of Language Model Alignment