mlsanityData to 14 Oct 2026

Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models

arXiv 2507.01915 ↗First posted 2 Jul 2025cs.CL cs.AI cs.LGcited by 23

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has emerged as a powerful technique for aligning large language models (LLMs) with human preferences. However, effectively aligning LLMs with diverse human preferences remains a significant challenge, particularly when they are conflict. To address this issue, we frame human value alignment as a multi-objective optimization problem, aiming to maximize a set of potentially conflicting objectives. We introduce Gradient-Adaptive Policy Optimization (GAPO), a novel fine-tuning paradigm that employs multiple-gradient descent to align LLMs with diverse preference distributions. GAPO adaptively rescales the gradients for each objective to determine an update direction that optimally balances the trade-offs between objectives. Additionally, we introduce P-GAPO, which incorporates user preferences across different objectives and achieves Pareto solutions that better align with the user's specific needs. Our theoretical analysis demonstrates that GAPO converges towards a Pareto optimal solution for multiple objectives. Empirical results on Mistral-7B show that GAPO outperforms current state-of-the-art methods, achieving superior performance in both helpfulness and harmlessness.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 48 references that are in the map. First posted on arXiv 2 Jul 2025.

Builds on

27 of 48 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization6 Dec
  2. 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models27 Aug
  3. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  4. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs3 Feb
  5. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  12. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
9 more
  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  3. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  4. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  5. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  6. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  7. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  8. 2023Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspectivecited by 51
  9. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 18.

23 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  3. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
  4. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 6
  5. 2026MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Freecited by 5
  6. 2025Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMscited by 4
  7. 2026UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignmentcited by 3
  8. 2026Ads in AI Chatbots? An Analysis of How Large Language Models Navigate Conflicts of Interestcited by 3
  9. 2026GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimizationcited by 3
  10. 2026Uncovering Cross-Objective Interference in Multi-Objective Alignmentcited by 2
  11. 2026Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimizationcited by 2
  12. 2026Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimizationcited by 2
11 more
  1. 2025FIRM: Federated In-client Regularized Multi-objective Alignment for Large Language Modelscited by 1
  2. 2026Palette: A Modular, Controllable, and Efficient Framework for On-demand Authorized Safety Alignment Relaxation in LLMscited by 1
  3. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  4. 2026Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RLcited by 1
  5. 2026Spillover-Aware Multi-Value Steering for Pluralistic LLM Alignmentcited by 1
  6. 2026MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignmentcited by 0
  7. 2026Chebyshev Center-Based Direction Selection for Multi-Objective Optimization and Training PINNscited by 0
  8. 2026Learn Before You Judge: Progressive Knowledge-to-Decision Alignment for Explainable Hateful Meme Detectioncited by 0
  9. 2026RMB: Reward Model Boosting Mitigates Reward Hackingcited by 0
  10. 2026ORPG: Reconciling Multiple Reward Objectives through Objective-wise Policy Gradientscited by 0
  11. 2026Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distributioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  2. 2025PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation
  3. 2024Towards Efficient Exact Optimization of Language Model Alignment
  4. 2025MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment
  5. 2025UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality
  6. 2024Accelerated Preference Optimization for Large Language Model Alignment
  7. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  8. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback