mlsanityData to 14 Oct 2026

C3: Rethinking Process Supervision via Confidence-Guided Credit Correction in Multi-Turn Reinforcement Learning

arXiv 2610.19695 ↗First posted 14 Oct 2026cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Large language models are increasingly trained as multi-turn agents that interact with external tools. However, outcome-only reinforcement learning can produce a form of misguided reinforcement: even on successful trajectories, undesirable local behaviors may receive positive token-level advantages and be favored by the policy update. To address this issue, we propose Confidence-Guided Credit Correction (C3), a plug-in method for local credit correction. C3 identifies spans corresponding to undesirable behaviors and selectively corrects positive credit after advantage estimation, with the correction strength adaptively controlled by the behavior policy's confidence in the corresponding behavior, while leaving the task reward and original credit estimation unchanged. Across multiple multi-turn benchmarks, C3 improves performance under both critic-based and critic-free reinforcement learning methods. Injecting the same behavioral signals as turn-level or outcome rewards does not consistently improve performance. Our results show that local credit correction provides an effective form of process supervision for outcome-driven RL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 35 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

31 of 35 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation12 May
  2. 2026Self-Induced Outcome Potential: Turn-Level Credit Assignment for Agents without Verifiers6 May
  3. 2026T2PO: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learning4 May
  4. 2026Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization15 Apr
  5. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization8 Jan
  6. 2025MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation11 Nov
  7. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  8. 2023Let's Verify Step by Step31 May
  9. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  6. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  7. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  8. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  10. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  11. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  12. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
10 more
  1. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  2. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 118
  3. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 88
  4. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agentscited by 48
  5. 2025Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMscited by 30
  6. 2026HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agentscited by 18
  7. 2026HISR: Hindsight Information Modulated Segmental Process Rewards For Multi-turn Agentic Reinforcement Learningcited by 8
  8. 2026LOTAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoningcited by 5
  9. 2026TRUST-SQL: Tool-Integrated Multi-Turn Reinforcement Learning for Text-to-SQL over Unknown Schemascited by 3
  10. 2026How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization
  2. 2025Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspective
  3. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
  4. 2026FARCA: Fact-Aligned Reliability-Aware Credit Assignment for Reinforcement Learning with Factual Supervision
  5. 2026AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning
  6. 2026PACT: From Credit Assignment to Critic Alignment
  7. 2026Internalizing Outcome Supervision into Process Supervision: A New Paradigm for Reinforcement Learning for Reasoning
  8. 2026Credit Assignment with Resets in Language Model Reasoning