When Correct Solutions Repeat: Rarity-Aware Credit Redistribution for GRPO
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Reinforcement learning with verifiable rewards (RLVR) com- monly optimizes each correct completion as an independent learning signal. In GRPO, this completion-level uniformity creates structure-level skew: recurring correct solution forms accumulate positive coefficient mass in proportion to how often they are sampled, while rare forms receive limited credit. We formalize this behavior as multiplicity-induced structure-level credit concentration and introduce a partition- conditioned rule that redistributes positive advantages accord- ing to cluster rarity. Cue-GRPO instantiates this rule with- out auxiliary-model inference by using deterministic Strategy Cues to construct rollout-local partitions of verified-correct traces. Across Qwen2.5-Math-7B and Llama-3.1-8B-Instruct, Cue-GRPO improves AIME repeated-sampling performance, with the largest gains at high sampling budgets. Credit Re- distribution (CR) under Judge Partitions (JP) further indi- cates that the proposed redistribution mechanism can oper- ate with judge-derived partitions. Cue-GRPO adds only 6% wall-clock training overhead over GRPO. These results sup- port structure-level credit redistribution as a practical design axis for RLVR, with Strategy Cues providing a low-overhead implementation for competition mathematics. Code is avail- able at https://github.com/CzZ12/When-Correct-Solutions- Repeat-Rarity-Aware-Credit-Redistribution-for-GRPO.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 32 references that are in the map. First posted on arXiv 4 Aug 2026.
Builds on
20 of 32 references.
Closest ancestors · 10
Same direction, newest first
- 2026Poly-EPO: Training Exploratory Reasoning Models19 Apr
- 2026F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare6 Feb
- 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning1 Feb
- 2026MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning30 Jan
- 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity29 Jul
- 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
- 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
- 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 10
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
- 2025Humanity's Last Examcited by 467
- 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
- 2025Beyond the Sampled Token: Preserving Candidate Support in RLVRcited by 18
- 2026Teacher-Free Self-Training Amplifies but Does Not Compound: A Pass@K Crossover on a Free-Verifier Domaincited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025GRPO is Secretly a Process Reward Model
- 2026Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards
- 2026Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distance
- 2026Stepwise Credit Assignment for GRPO on Flow-Matching Models
- 2026Hierarchical Credit Assignment for RLVR on Fused Gromov-Wasserstein Geometry
- 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification
- 2026ACPO: Asymmetric Credit Policy Optimization via Mode-Local Entropy Surrogate
- 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening