mlsanityData to 7 Oct 2026

ReCo: Reweighting GRPO Against Distributional Concentration

arXiv 2607.26862 ↗First posted 29 Jul 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO) has become a standard reinforcement learning method for post-training language models. Recent work shows that GRPO can reduce the base model's reasoning capacity and underperform it in Pass@k when k is large, indicating reduced coverage of reasoning paths. We find that this reduction is associated with GRPO concentrating on responses that the base model already generates with high probability. We trace this concentration to two mechanisms in the GRPO update. At the response level, high-probability responses dominate the group gradient through repeated occurrence. At the token level, GRPO's importance ratio scales gradients, further reinforcing tokens that become more likely under the current policy. We propose ReCo, a reweighting method that addresses both effects. Response contributions are normalized by their expected occurrence within the rollout group, and the token-level importance ratio is replaced with a variance-based ratio that gives larger update scale to non-saturated decision points where alternative token choices remain plausible. Across Qwen2.5-Math-1.5B/7B and Llama-3.1-8B-Instruct on five mathematical reasoning benchmarks, ReCo improves Pass@k for large values of k and is comparable to GRPO for small values of k.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 29 references that are in the map. First posted on arXiv 29 Jul 2026.

Builds on

23 of 29 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling25 Jun
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  3. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  4. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  5. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoning29 May
  6. 2025DRA-GRPO: Your GRPO Needs to Know Diverse Reasoning Paths for Mathematical Reasoning14 May
  7. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
  8. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  10. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan

Other references · 11

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2024OpenAI o1 System Cardcited by 1,220
  6. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  7. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
  8. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 92
  9. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
  10. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 36
  11. 2025When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RLcited by 21

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?
  2. 2025DARO: Difficulty-Aware Reweighting Policy Optimization
  3. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization
  4. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO
  5. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
  6. 2025λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences
  7. 2025TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback
  8. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning