mlsanityData to 14 Oct 2026

Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thought

arXiv 2604.17912 ↗First posted 20 Apr 2026cs.LG cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

State-of-the-art reasoning models utilize long chain-of-thought (CoT) to solve increasingly complex problems using more test-time computation. In this work, we explore a long CoT setting where the model makes up to K successive attempts at solving a problem, in which each attempt is allowed to build on earlier ones after the model receives a hard verifier feedback. This motivates RL methods that can harness per-attempt rewards by carefully weighting individual attempts. We study optimizing the Verification@K reward (the model succeeds by the K-th attempt) and show that naively weighing the attempts by their pass/fail results in biased gradients. We introduce Calibrated Attempt-Level (CAL) GRPO by devising a weighing strategy to obtain unbiased gradients while maintaining small variance. Our theory reveals how incorporating per-attempt rewards influence the training and the eventual Verification@K performance. Experiments, baselines, and ablations on synthetic and real data corroborate our theory and the benefits of CAL-GRPO over vanilla GRPO as well as naive weighting.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 53 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

44 of 53 references.

Closest ancestors · 17

Same direction, newest first

  1. 2026Asymmetric Prompt Weighting for Reinforcement Learning with Verifiable Rewards11 Feb
  2. 2025Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization18 Nov
  3. 2025MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation11 Nov
  4. 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients27 Oct
  5. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  6. 2025A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning18 Jul
  7. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  8. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment17 May
  9. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  10. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  11. 2025Process Reward Models That Think23 Apr
  12. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
5 more
  1. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  2. 2025Process Reinforcement through Implicit Rewards3 Feb
  3. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  4. 2023Let's Verify Step by Step31 May
  5. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  12. 2020Learning to summarize from human feedbackcited by 2,588
15 more
  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  3. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  4. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  5. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  6. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 324
  7. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  8. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 108
  9. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agentscited by 48
  10. 2024Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoningcited by 40
  11. 2025AlphaMaze: Enhancing Large Language Models' Spatial Intelligence via GRPOcited by 33
  12. 2023A Configurable Library for Generating and Manipulating Maze Datasetscited by 31
  13. 2025Multi-Turn Code Generation Through Single-Step Rewardscited by 28
  14. 2025SALT: Step-level Advantage Assignment for Long-horizon Agents via Trajectory Graphcited by 28
  15. 2025The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoningcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026VSPO: Vector-Steered Policy Optimization for Behavioral Controlcited by 0
  2. 2026Theoretical Foundations of max@k Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Stable Reinforcement Learning for Efficient Reasoning
  2. 2025Calibrated Reasoning: An Explanatory Verifier for Dynamic and Efficient Problem-Solving
  3. 2026RREDCoT: Segment-Level Reward Redistribution for Reasoning Models
  4. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
  5. 2025Dissecting Long-Chain-of-Thought Reasoning Models: An Empirical Study
  6. 2026SLPO: Scaling Latent Reasoning via a Surrogate Policy
  7. 2026Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoning
  8. 2026Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards