mlsanityData to 14 Oct 2026

GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization

arXiv 2601.05242 ↗First posted 8 Jan 2026cs.CL cs.AI cs.LGcited by 193

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

As language models become increasingly capable, users expect them to provide not only accurate responses but also behaviors aligned with diverse human preferences across a variety of scenarios. To achieve this, Reinforcement learning (RL) pipelines have begun incorporating multiple rewards, each capturing a distinct preference, to guide models toward these desired behaviors. However, recent work has defaulted to apply Group Relative Policy Optimization (GRPO) under multi-reward setting without examining its suitability. In this paper, we demonstrate that directly applying GRPO to normalize distinct rollout reward combinations causes them to collapse into identical advantage values, reducing the resolution of the training signal and resulting in suboptimal convergence and, in some cases, early training failure. We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues by decoupling the normalization of individual rewards, more faithfully preserving their relative differences and enabling more accurate multi-reward optimization, along with substantially improved training stability. We compare GDPO with GRPO across three tasks: tool calling, math reasoning, and coding reasoning, evaluating both correctness metrics (accuracy, bug ratio) and constraint adherence metrics (format, length). Across all settings, GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.

Key facts. Cited by 193 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 39 references that are in the map. First posted on arXiv 8 Jan 2026.

Builds on

32 of 39 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning13 Aug
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2025Training Language Models to Reason Efficiently6 Feb
  5. 2025Process Reinforcement through Implicit Rewards3 Feb
  6. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 25

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  8. 2021Measuring Coding Challenge Competence With APPScited by 902
  9. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  10. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  11. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 384
  12. 2025ToolRL: Reward is All Tool Learning Needscited by 362
13 more
  1. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  2. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruningcited by 279
  3. 2024ToolACE: Winning the Points of LLM Function Callingcited by 216
  4. 2023TACO: Topics in Algorithmic COde generation datasetcited by 160
  5. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 111
  6. 2024Rule Based Rewards for Language Model Safetycited by 108
  7. 2025Learn to Reason Efficiently with Adaptive Length-based Reward Shapingcited by 75
  8. 2025ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoningcited by 72
  9. 2024Hammer: Robust Function-Calling for On-Device Language Models via Function Maskingcited by 55
  10. 2025DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learningcited by 35
  11. 2025Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewardscited by 20
  12. 2024GenderAlign: An Alignment Dataset for Mitigating Gender Bias in Large Language Modelscited by 18
  13. 2024ALaRM: Align Language Models via Hierarchical Rewards Modelingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 193.

193 papers.

Sort:

By area:

  1. 2026Flow-OPD: On-Policy Distillation for Flow Matching Modelscited by 34
  2. 2026JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolutioncited by 16
  3. 2026Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligencecited by 15
  4. 2026Multi-Task GRPO: Reliable LLM Reasoning Across Taskscited by 14
  5. 2026DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editingcited by 14
  6. 2026JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligencecited by 14
  7. 2026Future-KL Regularized GRPO: Process-Level Credit Assignment from f-Divergence Regularizationcited by 12
  8. 2026CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMscited by 12
  9. 2026Flow-Factory: A Unified Framework for Reinforcement Learning in Flow-Matching Modelscited by 12
  10. 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 12
  11. 2026JoyAI-Image: Awaking Spatial Intelligence in Unified Multimodal Understanding and Generationcited by 12
  12. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 12
38 more
  1. 2026AeSlides: Incentivizing Aesthetic Layout in LLM-Based Slide Generation via Verifiable Rewardscited by 11
  2. 2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVRcited by 11
  3. 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Taskscited by 10
  4. 2026Multi-Faceted Interactivity Alignment in Full-Duplex Speech Modelscited by 10
  5. 2026When to Memorize and When to Stop: Gated Recurrent Memory for Long-Context Reasoningcited by 9
  6. 2026Socratic-SWE: Self-Evolving Coding Agents via Trace-Derived Agent Skillscited by 9
  7. 2026Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RLcited by 9
  8. 2026Learning Decentralized LLM Collaboration with Multi-Agent Actor Criticcited by 8
  9. 2026Vector Policy Optimization: Training for Diversity Improves Test-Time Searchcited by 8
  10. 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Securitycited by 8
  11. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  12. 2026Generative Recommendation for Large-Scale Advertisingcited by 7
  13. 2026Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibrationcited by 7
  14. 2026OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generationcited by 7
  15. 2026Tournament-GRPO: Group-Wise Tournament Rewards for Reinforcement Learning in Open-Ended Long-Form Generationcited by 7
  16. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 6
  17. 2026Faithful GRPO: Improving Visual Spatial Reasoning in Multimodal Language Models via Constrained Policy Optimizationcited by 6
  18. 2026Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Modelscited by 6
  19. 2026Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignmentcited by 6
  20. 2026TIR-Agent: Training an Explorative and Efficient Agent for Image Restorationcited by 5
  21. 2026Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPOcited by 5
  22. 2026Qwen-Image-2.0-RL Technical Reportcited by 5
  23. 2026The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Workscited by 5
  24. 2026What does RL improve for Visual Reasoning? A Frankenstein-Style Analysiscited by 4
  25. 2026On-Policy Supervised Fine-Tuning for Efficient Reasoningcited by 4
  26. 2026HAD: Combining Hierarchical Diffusion with Metric-Decoupled RL for End-to-End Drivingcited by 4
  27. 2026EvoGround: Self-Evolving Video Agents for Video Temporal Groundingcited by 4
  28. 2026Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMscited by 4
  29. 2026Discovering Process-Outcome Credit in Multi-Step LLM Reasoningcited by 3
  30. 2026TimeRFT: Stimulating Generalizable Time Series Forecasting for TSFMs via Reinforcement Finetuningcited by 3
  31. 2026RVPO: Risk-Sensitive Alignment via Variance Regularizationcited by 3
  32. 2026PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoningcited by 3
  33. 2026DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learningcited by 3
  34. 2026APE: Agentic Prompt Enhancer for Image Generation and Editingcited by 3
  35. 2026GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimizationcited by 3
  36. 2026DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Modelscited by 3
  37. 2026Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradientscited by 3
  38. 2026Scaling Multi-Reference Image Generation with Dynamic Reward Optimizationcited by 3

50 most cited of 193 All 193 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026GOPO: Policy Optimization using Ranked Rewards
  2. 2026MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning
  3. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization
  4. 2026Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment
  5. 2025BNPO: Beta Normalization Policy Optimization
  6. 2025Accelerating RLHF Training with Reward Variance Increase
  7. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  8. 2026Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients