mlsanityData to 14 Oct 2026

RREDCoT: Segment-Level Reward Redistribution for Reasoning Models

arXiv 2606.06475 ↗First posted 4 Jun 2026cs.LG cs.AIcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent advancements in reasoning language models have been driven by Reinforcement Learning (RL) fine-tuning. Most often, these rely on the Group Relative Policy Optimization (GRPO) algorithm or modifications thereof to steer the models to produce Chain-of-Thought (CoT) traces. The final answer can only be verified, and the reward assigned, after the CoT trace is complete, making it a delayed reward problem. GRPO and its modifications correspond to Monte Carlo methods in standard RL, which are known to suffer from high variance. A possible solution to this problem is the redistribution of rewards through credit assignment, where segments of the CoT trace that are important for arriving at the desirable solution are emphasized by assigning a higher reward. While Monte Carlo sampling can be used to provide an unbiased estimate of intermediate state values, its computational overhead makes it unsuitable for train-time credit assignment in long contexts at high granularity. We introduce RREDCoT (Reward REDistribution for Chain of Thoughts), which utilizes the model itself to approximate the optimal reward redistribution without additional generation. We investigate the advantages of our method compared to MC sampling and several attribution methods. We further analyze several aspects relevant to the construction of the redistribution such as segmentation of CoT traces and state value estimation.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 44 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

28 of 44 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026Beyond Uniform Credit: Causal Credit Assignment for Policy Optimization10 Feb
  2. 2026Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning12 Jan
  3. 2026Segmental Advantage Estimation: Enhancing PPO for Long-Context LLM Training12 Jan
  4. 2025CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment4 Aug
  5. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers23 Jun
  6. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  7. 2025BNPO: Beta Normalization Policy Optimization3 Jun
  8. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models29 May
  9. 2025Reinforcing General Reasoning without Verifiers27 May
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  11. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't20 Mar
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
3 more
  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 13

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025s1: Simple test-time scalingcited by 1,341
  4. 2025Group Sequence Policy Optimizationcited by 835
  5. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  6. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
  7. 2023Explaining Reinforcement Learning with Shapley Valuescited by 26
  8. 2024AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attributioncited by 23
  9. 2024Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measurecited by 18
  10. 2025Let's Verify Math Questions Step by Stepcited by 18
  11. 2025Compute as Teacher: Turning Inference Compute Into Reference-Free Supervisioncited by 18
  12. 2025Beyond the Last Answer: Your Reasoning Trace Uncovers More than You Thinkcited by 16
1 more
  1. 2025SERL: Self-Examining Reinforcement Learning on Open-Domaincited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flatteningcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025RM-R1: Reward Modeling as Reasoning
  2. 2026Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models
  3. 2025Stable Reinforcement Learning for Efficient Reasoning
  4. 2026Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thought
  5. 2025TreeRPO: Tree Relative Policy Optimization
  6. 2025Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL
  7. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening
  8. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective