mlsanityData to 14 Oct 2026

Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learning

arXiv 2609.35082 ↗First posted 28 Sep 2026cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Group-based reinforcement learning such as GRPO trains LLM agents by comparing rollouts sampled for each task, without a learned critic. In long-horizon settings, these rollouts revisit shared anchor states, offering cross-rollout evidence for step-level credit. Ideally, step-level credit should incorporate evidence beyond the realized suffixes observed at an anchor while aggregating alternative continuations according to their empirical frequencies. Visit-local averaging pools realized suffix returns at shared anchors and respects observed frequencies, but does not recursively propagate evidence across rollouts, whereas shortest-path estimators have global reach but allow a rarely observed route to dominate an anchor's value. We introduce Cross-Rollout Bellman Closure (CRBC), which merges each rollout group into a finite empirical process with absorbing success and failure boundaries and evaluates its behavior-policy Bellman fixed point with one linear solve. This fixed point uses the same empirical action and transition frequencies to propagate evidence through shared anchors and aggregate alternative continuations. Backing up the resulting state values through observed transitions yields action values, whose gain over the corresponding state value provides step-level credit. A corresponding finite-depth family recovers visit-local return averaging at zero depth and converges to the exact closure as depth increases. The normalized closure credit is combined with the trajectory-level group advantage for policy optimization, without additional environment rollouts. Across ALFWorld, WebShop, and Sokoban benchmarks with multiple model scales, CRBC consistently improves final performance and learning efficiency. For example, CRBC outperforms the strongest evaluated baseline by 5.59 percentage points on ALFWorld with Qwen2.5-1.5B-Instruct.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 48 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

42 of 48 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization15 Apr
  2. 2026RTMC: Step-Level Credit Assignment via Rollout Trees13 Apr
  3. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks26 Feb
  4. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  5. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  9. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2024Qwen2 Technical Reportcited by 2,120
  12. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
19 more
  1. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  2. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  4. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
  5. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  6. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  7. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  8. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  9. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
  10. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  11. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 108
  12. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 108
  13. 2026Hindsight Credit Assignment for Long-Horizon LLM Agentscited by 65
  14. 2026Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learningcited by 31
  15. 2026Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learningcited by 13
  16. 2026Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Taskscited by 9
  17. 2026CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilitiescited by 8
  18. 2026When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Trainingcited by 7
  19. 2026Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits
  2. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
  3. 2026Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regime
  4. 2026Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR
  5. 2026TIGPO: Temporal Instance-Graph Policy Optimization for Long-Horizon LLM Agents
  6. 2026RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning
  7. 2026VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learning
  8. 2026R2PO: Decoupling Rollout and Inference Policies for LLM Reasoning