# Save Your Saturated Data: Learning Beyond Reward Saturation in Group-Based RL

arXiv: https://arxiv.org/abs/2609.33126 · HTML: https://mlsanity.com/paper/2609.33126/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 24 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Builds on (most cited first, 22 of 22)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [R-Zero: Self-Evolving Reasoning LLM from Zero Data](https://mlsanity.com/paper/2508.05004/) (2025)
- [Generalizing Verifiable Instruction Following](https://mlsanity.com/paper/2507.02833/) (2025)
- [Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning](https://mlsanity.com/paper/2506.06632/) (2025)
- [No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping](https://mlsanity.com/paper/2509.21880/) (2025)
- [Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning](https://mlsanity.com/paper/2602.14338/) (2026)
- [RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning](https://mlsanity.com/paper/2606.01281/) (2026)
- [Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation](https://mlsanity.com/paper/2605.21125/) (2026)
- [Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information](https://mlsanity.com/paper/2607.19313/) (2026)
- [Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning](https://mlsanity.com/paper/2607.22002/) (2026)
- [Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data](https://mlsanity.com/paper/2604.18493/) (2026)
- [Question Begets Question: Self-Evolving Curriculum for Reinforcement Fine-Tuning on Competition Mathematics](https://mlsanity.com/paper/2608.01522/) (2026)
- [Prism-GRPO: Faster VLA Policy Optimization via Splitting Same-outcome Groups](https://mlsanity.com/paper/2608.17423/) (2026)

## Abstract (arXiv)

Group-relative reinforcement learning (RL) relies on reward variation among sampled responses to estimate informative relative advantages. As language models become increasingly capable, existing training data can become reward-saturated: all sampled responses to the same problem might receive equally high rewards, where the group-relative learning signals vanish and leave previously useful data obsolete. In this work, we investigate whether useful learning signals can be recovered from such saturated data. We study interventions at four levels of group-policy RL pipelines---data, rollout, reward, and advantage---and conduct extensive RL training on saturated reasoning data only. While standard GRPO on saturated data would almost always yield near-0 advantages and near-noise signals, diverse interventions successfully recycle and repurpose such data: among the proposed strategies, interventions at rollout generation are consistently most effective: nudging the policy to generate ``high-quality'', incorrect solutions introduces rollouts with poor rewards into saturated groups as negative samples, which turns out to improve GRPO by 6.4% to 9.0% across Qwen3-1.7B and 4B. Other interventions such as increasing rollout temperature or adding auxiliary rewards can also restore non-zero advantages, but yield less consistent gains. Further analyses show that effective negative rollouts require informative negative trajectories, that the method remains effective alongside unsaturated data, and that it supports iterative recycling of newly saturated examples. While increasingly stronger LLMs would render more data as saturated, our results demonstrate that don't waste your saturated data: with the right strategies they can be recycled into useful RL training signals in an increasingly data-scarce world.