mlsanityData to 14 Oct 2026

Let Credit Follow Computation: Architecture-Aware Credit Transport for Large Language Model Reinforcement Learning

arXiv 2608.21501 ↗First posted 21 Aug 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Credit assignment in large-language-model reinforcement learning (LLM RL) can be separated into three objects: evidence about success, a transport operator that converts this evidence into token-level advantages, and an update geometry that turns advantages into policy changes. Recent work has greatly improved evidence, sampling, and update geometry, but the transport operator is usually architecture-agnostic. Fixed-discount GAE applies a stationary geometric kernel along token time; group-relative methods broadcast an outcome statistic across an entire response. Neither operator represents the trajectory-specific computation used by the Transformer policy itself. We introduce computation-conditioned credit transport (CCT), a general framework in which a detached statistic of the behavior policy's internal computation parameterizes the causal kernel that transports downstream value through a rollout. Our concrete algorithm, CompPO, maps native attention concentration to a bounded per-token retention gate, uses the gate in both the one-step bootstrap and a path-dependent generalized-advantage trace (Comp-GAE), and co-designs a transport-aligned critic (TAC) that reuses the actor's hidden states and routing information without a second same-scale Transformer. The task reward and clipped PPO policy objective remain unchanged; a constant gate recovers fixed-coefficient GAE. Across five Qwen3-4B seeds, CompPO reaches 61.4% final held-out accuracy (95% CI [60.8,62.0]) versus 53.8% [52.9,54.7] for tuned GRPO. Neither Comp-GAE with a standard critic (55.2%) nor TAC with a fixed gate (56.4%) matches the full model (interaction +2.4 [1.9,2.9]). Shuffle and position controls confirm trajectory-specific alignment; CompPO is stable in 10/12 PPO-grid runs versus 3/12. Frozen evaluation improves over GRPO by 4.3 and 3.9 greedy pass@1 macro points on Qwen3-4B and Llama-3.1-8B-Instruct.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 21 Aug 2026.

Builds on

30 of 40 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
18 more
  1. 2019Attention is not Explanationcited by 929
  2. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  3. 2025Group Sequence Policy Optimizationcited by 835
  4. 2019Attention is not not Explanationcited by 636
  5. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  6. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secretcited by 98
  7. 2016Unifying task specification in reinforcement learningcited by 41
  8. 2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMscited by 27
  9. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 25
  10. 2025GRPO-λ: Credit Assignment improves LLM Reasoningcited by 23
  11. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Modelscited by 23
  12. 2026Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoningcited by 19
  13. 2026Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learningcited by 11
  14. 2025Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoningcited by 10
  15. 2026Reinforced Attention Learningcited by 6
  16. 2026Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVRcited by 5
  17. 2026How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMscited by 3
  18. 2026AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PACT: From Credit Assignment to Critic Alignment
  2. 2026Contextual Credit Policy Optimization
  3. 2026Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning
  4. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models
  5. 2026ArenaFlow: From Trajectory Ranking to Hierarchical Credit Propagation for Open-Ended Agent RL
  6. 2024Assessing the Zero-Shot Capabilities of LLMs for Action Evaluation in RL
  7. 2026How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learning
  8. 2026S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF