mlsanityData to 7 Oct 2026

CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning

arXiv 2609.36820 ↗First posted 29 Sep 2026cs.LG cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO) is widely used to train reasoning language models, where it computes advantages by centering and normalizing rewards across rollouts of the same prompt. For multiple rewards, GRPO sums the reward components and normalizes the total reward by its within-group standard deviation. The corresponding variance equals the sum of all pairwise reward covariances. For a fixed centered reward, larger aggregate covariance produces smaller advantages, and vice versa, allowing update magnitudes to adapt to reward dependence. However, correlated rewards with large scales can dominate this normalization and suppress signals from smaller-scale rewards. We propose Correlation-Normalized GRPO (CorrGRPO), which normalizes pairwise covariances into Pearson correlation coefficients. CorrGRPO keeps the centered total reward unchanged while balancing the influence of differently scaled rewards on the correlation-based normalization. This allows advantage magnitudes to adapt to reward correlations without the normalization being dominated by large-scale reward components. We compare CorrGRPO with GRPO and other variants on code generation, tool calling, and agent security, using models ranging from 0.5B to 8B parameters. These tasks all involve multiple rewards that can improve together or present tradeoffs. Results show improvements across three domains, including code generation, tool calling, and agent security. Our code is available at https://github.com/HKUST-KnowComp/CorrGRPO.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 26 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

24 of 26 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization8 Jan
  2. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems26 Sep
  3. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan

Other references · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  8. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  9. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  10. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  11. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  12. 2025ToolRL: Reward is All Tool Learning Needscited by 342
7 more
  1. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  2. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  3. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  4. 2025LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMscited by 67
  5. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  6. 2024Multi-objective Reinforcement learning from AI Feedbackcited by 9
  7. 2026Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimizationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
  2. 2026Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervals
  3. 2026When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO
  4. 2026AMIR-GRPO: Inducing Implicit Preference Signals into GRPO
  5. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization
  6. 2026iGRPO: Self-Feedback-Driven LLM Reasoning
  7. 2026GOPO: Policy Optimization using Ranked Rewards
  8. 2026Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients