mlsanityData to 7 Oct 2026

Robust Multi-Objective Preference Alignment with Online DPO

arXiv 2503.00295 ↗First posted 1 Mar 2025cs.CL cs.LGcited by 21

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Multi-objective preference alignment of large language models (LLMs) is critical for developing AI systems that are more configurable, personalizable, helpful, and safe. However, optimizing model outputs to satisfy diverse objectives with variable weights at inference time for truly personalized models presents a significant challenge. Existing approaches are either computationally expensive to train or do not sufficiently steer model behaviors. This paper introduces the Multi-Objective Online DPO (MO-ODPO) algorithm, designed to robustly and efficiently align model behaviors with multiple, potentially conflicting human preferences. Our approach incorporates a prompt conditioning mechanism, allowing us to train a single preference-conditional policy, that can adapt to new preference combinations at inference. Experiments on two popular benchmarks show that MO-ODPO Pareto-dominates existing baselines while providing excellent inference-time steerability between diverse objectives.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 43 references that are in the map. First posted on arXiv 1 Mar 2025.

Builds on

33 of 43 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Decoding-Time Language Model Alignment with Multiple Objectives27 Jun
  2. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  3. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  4. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  5. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  6. 2024Decoding-time Realignment of Language Models5 Feb
  7. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  9. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  10. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  8. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  9. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  10. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  11. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  12. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
11 more
  1. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  3. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  4. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  5. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  6. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 107
  7. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 87
  8. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  9. 2023Click: Controllable Text Generation with Sequence Likelihood Contrastive Learningcited by 41
  10. 2023Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferencescited by 25
  11. 2019AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learningcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 13.

21 papers.

Sort:

By area:

  1. 2025CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectivescited by 10
  2. 2026Reward-free Alignment for Conflicting Objectivescited by 10
  3. 2025OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignmentcited by 7
  4. 2025Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolationcited by 5
  5. 2025SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimizationcited by 3
  6. 2026UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignmentcited by 3
  7. 2026Property-driven Protein Inverse Folding With Multi-Objective Preference Alignmentcited by 3
  8. 2025PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimizationcited by 2
  9. 2025Lightweight Robust Direct Preference Optimizationcited by 2
  10. 2025Aligning Machiavellian Agents: Behavior Steering via Test-Time Policy Shapingcited by 2
  11. 2026VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Modelscited by 2
  12. 2026Beyond Compromise: Pareto-Lenient Consensus for Efficient Multi-Preference LLM Alignmentcited by 2
9 more
  1. 2026Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansioncited by 1
  2. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  3. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selectioncited by 0
  4. 2026Many Preferences, Few Policies: Towards Scalable Language Model Personalizationcited by 0
  5. 2026MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignmentcited by 0
  6. 2026Multi-Objective Exploration and Preference Optimization via Mutual Informationcited by 0
  7. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0
  8. 2026Evolutionary Soups: Evolving Mixture-of-Experts for Multi-Objective LLM Alignmentcited by 0
  9. 2026PoEM: Predicting RL Outcomes from Existing Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models
  2. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  3. 2025Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
  4. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization
  5. 2025SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment
  6. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards
  7. 2024Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization
  8. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs