mlsanityData to 7 Oct 2026

TCPO: Turn-Level Credit Policy Optimization

arXiv 2608.01667 ↗First posted 3 Aug 2026cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Verifier-guided reinforcement learning has become a powerful paradigm for improving LLM reasoning. In multi-turn settings, models receive a verifier score after each turn and iteratively refine their outputs. Although such scores provide dense feedback, they do not directly provide dense credit: a score measures the quality of the current output, while credit should measure how the current turn changes the refinement trajectory. We propose TCPO, a turn-level credit assignment method for verifier-guided multi-turn RL. TCPO casts credit assignment as score-to-credit conversion and constructs turn-level advantages through reference-based comparisons: retrospective credit captures immediate progress and regression relative to the best prior state; hindsight delayed credit identifies non-improving turns with later payoff; and selective fixed-history counterfactual estimation refines high-surprisal turns under the same history. Experiments on math reasoning, code generation, and AppWorld agent tasks show that TCPO improves or matches the strongest baselines across model scales, task domains, and verifier types. TCPO achieves the best or tied-best best-turn Pass@8 on Qwen3-4B and DeepSeek-R1-Distill-Llama-8B, reduces turns to success, and improves multi-turn agent performance. These results highlight score-to-credit conversion as a central ingredient for verifier-guided multi-turn policy optimization.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 31 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

24 of 31 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  2. 2023Let's Verify Step by Step31 May
  3. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 21

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  4. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  5. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  6. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  7. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  8. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  9. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  10. 2025ToRL: Scaling Tool-Integrated RLcited by 166
  11. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  12. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 115
9 more
  1. 2020Keep CALM and Explore: Language Models for Action Generation in Text-based Gamescited by 108
  2. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  3. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learningcited by 107
  4. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 103
  5. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoningcited by 67
  6. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agentscited by 43
  7. 2025Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMscited by 28
  8. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agentscited by 11
  9. 2026AT2PO: Agentic Turn-based Policy Optimization via Tree Searchcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SkillGate: Training In-Policy Skill Selection in Long-Horizon Agentscited by 1
  2. 2026Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Usecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment
  2. 2026VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents
  3. 2026Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents
  4. 2025MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation
  5. 2026Coverage, Not Targeting: A Structural Regime in Multi-Turn Agent Credit Assignment
  6. 2025Pinpointing crucial steps: Attribution-based Credit Assignment for Verifiable Reinforcement Learning
  7. 2026ACPO: Asymmetric Credit Policy Optimization via Mode-Local Entropy Surrogate
  8. 2026RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training