# Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimation

arXiv: https://arxiv.org/abs/2609.36944 · HTML: https://mlsanity.com/paper/2609.36944/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 19 of 19)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [A General and Adaptive Robust Loss Function](https://mlsanity.com/paper/1701.03077/) (2017)
- [Evaluating the Performance of Large Language Models on GAOKAO Benchmark](https://mlsanity.com/paper/2305.12474/) (2023)
- [Your Group-Relative Advantage Is Biased](https://mlsanity.com/paper/2601.08521/) (2026)
- [Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification](https://mlsanity.com/paper/2407.14503/) (2024)
- [LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation](https://mlsanity.com/paper/2309.12455/) (2023)
- [RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training](https://mlsanity.com/paper/2510.00911/) (2025)
- [Variance Reduced Median-of-Means Estimator for Byzantine-Robust Distributed Inference](https://mlsanity.com/paper/2103.02860/) (2021)
- [Robust Kernel Density Estimation with Median-of-Means principle](https://mlsanity.com/paper/2006.16590/) (2020)
- [Mind the Gap: Norm-Aware Adaptive Robust Loss for Multivariate Least-Squares Problems](https://mlsanity.com/paper/2206.09215/) (2022)
- [An Adaptive Graduated Nonconvexity Loss Function for Robust Nonlinear Least Squares Solutions](https://mlsanity.com/paper/2305.06869/) (2023)

## Abstract (arXiv)

Group-relative policy optimization relies on reward-derived advantages and sequence-level likelihood weights, both of which can be sensitive to localized outliers. Extreme rewards can collapse the contrast among clean responses after group normalization, while token-level log-ratio perturbations can alter sequence weights and clipping decisions. We introduce RoVR-GSPO, a dual-channel robust optimizer that addresses these failure modes separately. Its reward channel combines robust reference estimation with bounded residual credit, while its ratio channel uses differentiable SoftRoVR aggregation to construct robust sequence weights. We provide stability and efficiency analyses for both channels. Experiments on mathematical reasoning, long-context summarization, and tool-call annotation show consistent improvements over GSPO, while controlled perturbation studies demonstrate stronger robustness to reward contamination and token-ratio anomalies.