# Leveraging Error Diversity in Group Rollouts for Reinforcement Learning

arXiv: https://arxiv.org/abs/2605.17333 · HTML: https://mlsanity.com/paper/2605.17333/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 17 May 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 25 of 25)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [MathArena: Evaluating LLMs on Uncontaminated Math Competitions](https://mlsanity.com/paper/2505.23281/) (2025)
- [Reasoning with Exploration: An Entropy Perspective](https://mlsanity.com/paper/2506.14758/) (2025)
- [CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings](https://mlsanity.com/paper/2501.01257/) (2025)
- [Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems](https://mlsanity.com/paper/2505.15201/) (2025)
- [EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity](https://mlsanity.com/paper/2507.21848/) (2025)
- [Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning](https://mlsanity.com/paper/2510.10649/) (2025)
- [Interleaved Latent Visual Reasoning with Selective Perceptual Modeling](https://mlsanity.com/paper/2512.05665/) (2025)
- [NGRPO: Negative-enhanced Group Relative Policy Optimization](https://mlsanity.com/paper/2509.18851/) (2025)
- [Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation](https://mlsanity.com/paper/2601.20614/) (2026)
- [SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning](https://mlsanity.com/paper/2602.01062/) (2026)
- [Reinforced Efficient Reasoning via Semantically Diverse Exploration](https://mlsanity.com/paper/2601.05053/) (2026)
- [Reasoning through Exploration: A Reinforcement Learning Framework for Robust Function Calling](https://mlsanity.com/paper/2508.05118/) (2025)
- [RSPO: Risk-Seeking Policy Optimization for Pass@k and Max@k Metrics in Large Language Models](https://mlsanity.com/paper/2508.01174/) (2025)
- [Grouter: Decoupling Routing from Representation for Accelerated MoE Training](https://mlsanity.com/paper/2603.06626/) (2026)
- [Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectories](https://mlsanity.com/paper/2511.07368/) (2025)
- [MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweighting](https://mlsanity.com/paper/2601.09085/) (2026)

## Built on by (most cited first, 3 of 3)

- [Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients](https://mlsanity.com/paper/2606.18216/) (2026)
- [Groupwise Agentic Grading and Advantage Redistribution for Code Agent RL](https://mlsanity.com/paper/2609.32577/) (2026)
- [Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning](https://mlsanity.com/paper/2606.03234/) (2026)

## Abstract (arXiv)

Reinforcement Learning from Verifiable Rewards (RLVR) typically samples multiple responses per prompt and assigns binary rewards based on individual correctness, yet the collective structure of the group output, specifically the distribution of errors, is largely discarded. We identify this as a missed opportunity: empirical analysis reveals that error diversity within a group is a strong predictor of training success, with problems eliciting diverse wrong answers benefiting substantially more from RLVR than those producing homogeneous failures. Motivated by this observation, we propose Error Diversity Advantage Shaping (EDAS), a lightweight, algorithm-agnostic technique that modulates the advantage signal for incorrect rollouts based on intra-group error diversity. EDAS amplifies penalties for dominant, repeated errors and attenuates penalties for rare, exploratory ones, thereby encouraging the model to maintain diverse reasoning paths and discouraging error perseveration. Crucially, EDAS operates as a simple post-hoc adjustment that can be seamlessly integrated into any RLVR algorithm. We validate EDAS on top of several mainstream RLVR methods across a series of models and seven challenging math benchmarks, demonstrating consistent improvements. Notably, EDAS yields an average improvement of 6.29 points over DAPO on Qwen3-8B across seven benchmarks, confirming that exploiting the latent information in group rollouts is a broadly effective strategy for strengthening RLVR.