mlsanityData to 14 Oct 2026

Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimation

arXiv 2609.36944 ↗First posted 29 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Group-relative policy optimization relies on reward-derived advantages and sequence-level likelihood weights, both of which can be sensitive to localized outliers. Extreme rewards can collapse the contrast among clean responses after group normalization, while token-level log-ratio perturbations can alter sequence weights and clipping decisions. We introduce RoVR-GSPO, a dual-channel robust optimizer that addresses these failure modes separately. Its reward channel combines robust reference estimation with bounded residual credit, while its ratio channel uses differentiable SoftRoVR aggregation to construct robust sequence weights. We provide stability and efficiency analyses for both channels. Experiments on mathematical reasoning, long-context summarization, and tool-call annotation show consistent improvements over GSPO, while controlled perturbation studies demonstrate stronger robustness to reward contamination and token-ratio anomalies.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 24 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

19 of 24 references.

References · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  7. 2025Group Sequence Policy Optimizationcited by 835
  8. 2021Efficient Attentions for Long Document Summarizationcited by 345
  9. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  10. 2017A General and Adaptive Robust Loss Functioncited by 236
  11. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 186
  12. 2026Your Group-Relative Advantage Is Biasedcited by 34
7 more
  1. 2024Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecificationcited by 23
  2. 2023LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisationcited by 16
  3. 2025RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Trainingcited by 12
  4. 2021Variance Reduced Median-of-Means Estimator for Byzantine-Robust Distributed Inferencecited by 11
  5. 2020Robust Kernel Density Estimation with Median-of-Means principlecited by 9
  6. 2022Mind the Gap: Norm-Aware Adaptive Robust Loss for Multivariate Least-Squares Problemscited by 5
  7. 2023An Adaptive Graduated Nonconvexity Loss Function for Robust Nonlinear Least Squares Solutionscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Constrained Group Relative Policy Optimization
  2. 2026STAR-GRPO: Canonical Anchoring and Reliability-First Advantages against Representation-Dependent Reward Hacking
  3. 2026On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse
  4. 2025NGRPO: Negative-enhanced Group Relative Policy Optimization
  5. 2026Blockwise Advantage Estimation for Multi-Objective RL with Verifiable Rewards
  6. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients
  7. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
  8. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective