PACT: From Credit Assignment to Critic Alignment
Not on the map. Why?
Abstract (arXiv)
Reinforcement learning has become a central component of large language model (LLM) post-training, yet token-level credit lacks a generally accepted mathematical definition, leaving its relationship to commonly used training signals unclear. We formulate three regularity conditions, namely Completeness, Prefix Consistency, and Neutrality, and prove that they uniquely determine token-level credit. This characterization provides a unified basis for explaining phenomena across existing algorithms and guides the development of an improved actor-critic training procedure. Through this lens, an ideal teacher in On-Policy Distillation (OPD) acts as an implicit critic, yielding an expected policy gradient proportional to that induced by token-level credit. Response-level REINFORCE Leave-One-Out (RLOO) signals match the expected policy-gradient contribution of token-level credit despite their coarser granularity. We further establish approximate credit sparsity under bounded outcome rewards and show how intermediate critic errors in Generalized Advantage Estimation (GAE) can become comparable to the underlying credit. These motivate Policy Aligned Critic Training (PACT), which adopts an Actor-then-Critic update order to apply importance sampling correction to critic training and better align the critic with the updated policy. In agentic mathematical reasoning, PACT achieves 72.87% average accuracy across four benchmarks, outperforming GRPO and PPO by 8.80 and 13.16 percentage points, respectively. On SWE-bench Verified, PACT achieves a pass rate of 67.4%, outperforming PPO, GRPO, and SAO by 2.4, 2.0, and 3.8 percentage points, respectively.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 22 Sep 2026.
Builds on
36 of 50 references.
References · 36
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
- 2020Learning to summarize from human feedbackcited by 2,562
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
- 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
- 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
- 2015Policy Distillationcited by 414
- 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
24 more
- 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 281
- 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
- 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
- 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
- 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
- 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
- 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 233
- 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
- 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
- 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
- 2018RUDDER: Return Decomposition for Delayed Rewardscited by 162
- 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
- 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
- 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 132
- 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secretcited by 95
- 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspectivecited by 75
- 2019Hindsight Credit Assignmentcited by 74
- 2020Counterfactual Credit Assignment in Model-Free Reinforcement Learningcited by 60
- 2018Optimizing Agent Behavior over Long Time Scales by Transporting Valuecited by 57
- 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 56
- 2025Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewardscited by 41
- 2025Toward Training Superintelligent Software Agents through Self-Play SWE-RLcited by 37
- 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learningcited by 26
- 2023Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysiscited by 10
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026ARCA: Adapter-Residual Credit Assignment When Token Signals Degenerate
- 2026T5: Twin-Critic Training for Token-Level Thoughts in Reinforcement Mid-Training
- 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models
- 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models
- 2026VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents
- 2026Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
- 2026Learning with a Single Rollout via Monte Carlo Pass@k Critic
- 2026Let Credit Follow Computation: Architecture-Aware Credit Transport for Large Language Model Reinforcement Learning