mlsanityData to 7 Oct 2026

MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignment

arXiv 2604.20685 ↗First posted 22 Apr 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models (LLMs) to desirable human values requires balancing multiple, potentially conflicting objectives such as helpfulness, truthfulness, and harmlessness, which presents a multi-objective optimisation challenge. Most alignment pipelines rely on a fixed scalarisation of these objectives, which can introduce procedural unfairness by systematically under-weighting harder-to-optimise or minority objectives. To promote more equitable trade-offs, we introduce MGDA-Decoupled, a geometry-based multi-objective optimisation algorithm that finds a shared descent direction while explicitly accounting for each objective's convergence dynamics. In contrast to prior methods that depend on reinforcement learning (e.g., GAPO) or explicit reward models (e.g., MODPO), our approach operates entirely within the lightweight Direct Preference Optimisation (DPO) paradigm. Experiments on the UltraFeedback dataset show that geometry-aware methods -- and MGDA-Decoupled in particular -- achieve the highest win rates against golden responses, both overall and per objective.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 44 references that are in the map. First posted on arXiv 22 Apr 2026.

Builds on

32 of 44 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models2 Jul
  2. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models8 Jun
  3. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models16 May
  4. 2025Robust Multi-Objective Preference Alignment with Online DPO1 Mar
  5. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization5 Oct
  6. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models25 Mar
  7. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  8. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  9. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  10. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs3 Feb
  11. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  12. 2023Mitigating the Alignment Tax of RLHF12 Sep
2 more
  1. 2022Training language models to follow instructions with human feedback4 Mar
  2. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 18

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  7. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  8. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  9. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  10. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  11. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  12. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
6 more
  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  2. 2025Qwen2.5-1M Technical Reportcited by 158
  3. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  4. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  5. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safetycited by 37
  6. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 15

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward-free Alignment for Conflicting Objectives
  2. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  3. 2025Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information
  4. 2025AlphaPO: Reward Shape Matters for LLM Alignment
  5. 2025Provably Convergent Primal-Dual DPO for Constrained LLM Alignment
  6. 2025Pareto Multi-Objective Alignment for Language Models
  7. 2025Enhancing Small LLM Alignment through Margin-Based Objective Modifications under Resource Constraints
  8. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards