mlsanityData to 7 Oct 2026

GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization

arXiv 2606.16771 ↗First posted 15 Jun 2026cs.LGcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

As LLMs advance, post-training reinforcement learning (RL) increasingly relies on multi-dimensional rewards to cultivate comprehensive capabilities. This shift demands new algorithms capable of optimizing diverse and potentially competing objectives simultaneously. To address this, existing methods such as Group reward-Decoupled Policy Optimization (GDPO) decompose the overall score into independent reward groups, then compute the RL loss separately within each group. However, this strategy still encounters multi-reward conflicts: a single rollout can yield positive advantages on certain reward dimensions but negative ones on others, causing opposing signals to cancel each other out during aggregation, further hindering RL training efficiency. Inspired by Dynamic sAmpling Policy Optimization (DAPO), which improves RL training efficiency by filtering out ineffective rollouts with near-zero advantages, we propose Group-Dynamic reward-Decoupled Policy Optimization (GD2PO). Specifically, GD2PO employs a conflict-aware filtering mechanism to mask out rollouts suffering from severe reward-wise disagreement. By preventing conflicting signals from canceling each other out, this masking strategy preserves and enhances the magnitude of effective RL advantages, thereby significantly accelerating learning efficiency. Furthermore, we introduce query-level reweighting to dynamically adjust the update intensity of each query based on its overall reward consensus. Experiments on various multi-reward scenarios, including tool calling and human preference alignment, demonstrate that GD2PO consistently and significantly outperforms existing baselines. The code is available at https://github.com/Qwen-Applications/GD2PO.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 54 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

47 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill2 Jun
  2. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization8 Jan
  3. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems26 Sep

Other references · 44

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  9. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  11. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  12. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
32 more
  1. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  2. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  3. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  4. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  5. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  6. 2024ToolACE: Winning the Points of LLM Function Callingcited by 204
  7. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  8. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 121
  9. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 108
  10. 2024Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuningcited by 87
  11. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 80
  12. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoningcited by 75
  13. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  14. 2024Hammer: Robust Function-Calling for On-Device Language Models via Function Maskingcited by 53
  15. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Modelscited by 43
  16. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 42
  17. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 40
  18. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 31
  19. 2023On Diversified Preferences of Large Language Model Alignmentcited by 26
  20. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  21. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Modelscited by 22
  22. 2025Pareto Multi-Objective Alignment for Language Modelscited by 20
  23. 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignmentcited by 14
  24. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Modelscited by 13
  25. 2024ALaRM: Align Language Models via Hierarchical Rewards Modelingcited by 11
  26. 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Modelcited by 10
  27. 2024Atoxia: Red-teaming Large Language Models with Target Toxic Answerscited by 7
  28. 2025Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidancecited by 7
  29. 2026CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVRcited by 5
  30. 2026MARCH: Multi-Agent Reinforced Self-Check for LLM Hallucinationcited by 5
  31. 2024Conflict-Averse Gradient Aggregation for Constrained Multi-Objective Reinforcement Learningcited by 4
  32. 2026Blockwise Advantage Estimation for Multi-Objective RL with Verifiable Rewardscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimizationcited by 2
  2. 2026MatrixReward: Reward from Rubric Matrix for Open-Ended Generationcited by 0
  3. 2026Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Breaking textitWinner-Takes-All: Cooperative Policy Optimization Improves Diverse LLM Reasoning
  2. 2026GOPO: Policy Optimization using Ranked Rewards
  3. 2026MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following
  4. 2026Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness
  5. 2026SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation
  6. 2026DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning
  7. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training
  8. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing