mlsanityData to 7 Oct 2026

RTMC: Step-Level Credit Assignment via Rollout Trees

arXiv 2604.11037 ↗First posted 13 Apr 2026cs.LG cs.AIcited by 9

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Multi-step agentic reinforcement learning benefits from fine-grained credit assignment, yet existing approaches offer limited options: critic-free methods like GRPO assign a uniform advantage to every action in a trajectory, while learned value networks introduce notable overhead and can be fragile under sparse rewards. We observe that group rollouts targeting the same problem often traverse overlapping intermediate states, implicitly forming a tree whose branches diverge at successive decision points. Building on this insight, we introduce Rollout-Tree Monte Carlo (RTMC) advantage estimation, which aggregates return statistics across rollouts sharing a common state to produce per-step Q-values and advantages--without any learned critic. A state-action signature system compresses raw interaction histories into compact, comparable representations, making cross-rollout state matching tractable. On SWE-bench Verified, RTMC improves pass@1 by 3.2 percentage points over GRPO.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 12 of its 24 references that are in the map. First posted on arXiv 13 Apr 2026.

Builds on

12 of 24 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMs22 Sep
  2. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 10

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  7. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 140
  8. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 58
  9. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 45
  10. 2025Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystemcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agentscited by 1
  2. 2026How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learningcited by 1
  3. 2026Back to the Definition: Estimating Step-Level Advantages via Trajectory Graphs for Agentic Reinforcement Learningcited by 0
  4. 2026SWE-MILE: Asynchronous Potential-Induced Milestone Credit Assignment for Long-Horizon Software Engineering Agentscited by 0
  5. 2026SkillRubric: Co-Evolving Actor Guidance and Evaluator Rubrics for Multimodal Agentscited by 0
  6. 2026Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learningcited by 0
  7. 2026EvoSteer: Online Self-Evolving Graph Orchestration via Reference-Anchored Credit Assignmentcited by 0
  8. 2026Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimizationcited by 0
  9. 2026Characterizing a Configuration Where Inference-Time PRM-Pruned Fragment Grafting Is Inert: Evidence from Three Reasoning LMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Process Reward Informed Tree Rollout for Effective Multi-Turn RL
  2. 2026TIGPO: Temporal Instance-Graph Policy Optimization for Long-Horizon LLM Agents
  3. 2026Information Gain-based Rollout Policy Optimization: An Adaptive Tree-Structured Rollout Approach for Multi-Turn LLM Agents
  4. 2026Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning
  5. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
  6. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning
  7. 2025Graph-Enhanced Policy Optimization in LLM Agent Training
  8. 2026StepOPSD: Step-Aware Online Preference Self-Distillation for Agent Reinforcement Learning