mlsanityData to 14 Oct 2026

Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPO

arXiv 2509.24494 ↗First posted 29 Sep 2025cs.CLcited by 10

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO) trains Chain-of-Thought reasoning with verifiable rewards, but estimating thought-level advantages without value functions often suffers from high variance. Although tree-style branching is used in practice to reduce variance, it lacks a theoretical explanation of why it works and whether it is important or potentially necessary. We study thought-level advantage estimation in GRPO from a variance perspective under a minimal tree-style setting where multiple continuations are sampled for each thought. Using the multivariate delta method, we reveal a sampling-dimension asymmetry. Increasing sampled thoughts (K) leaves a strictly positive estimation-variance floor, whereas increasing continuations per thought (M) drives the leading-order estimation variance to zero at rate 1/M. This implies that, within the fixed-temperature GRPO-style estimator without value models studied here, accurate thought-level advantage estimation cannot be achieved by scaling thought sampling alone, making continuation-level branching a principled and potentially necessary mechanism rather than a heuristic. Experiments further provide empirical evidence for its effectiveness and potential necessity, demonstrating improved optimization stability, training efficiency, and final performance not only in math but also across vision domains and under different model architectures and sizes.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 40 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

24 of 40 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling24 Aug
  2. 2025GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control5 Aug
  3. 2025Geometric-Mean Policy Optimization28 Jul
  4. 2025TreeRPO: Tree Relative Policy Optimization5 Jun
  5. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models29 May
  6. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  9. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 15

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2025Group Sequence Policy Optimizationcited by 835
  6. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  7. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
  8. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
  9. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  10. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  11. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 80
  12. 2025GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoningcited by 47
3 more
  1. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 28
  2. 2025ManipLVM-R1: Reinforcement Learning for Reasoning in Embodied Manipulation with Large Vision-Language Modelscited by 27
  3. 2025Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Roboticscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 8.

10 papers.

  1. 2025CogniEdit: Dense Gradient Flow Optimization for Fine-Grained Image Editingcited by 2
  2. 2026Agri-R1: Agricultural Reasoning for Disease Diagnosis via Automated-Synthesis and Reinforcement Learningcited by 1
  3. 2026RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solutioncited by 1
  4. 2026Latent Thought Credit: Multi-Answer Credit Assignment for Latent Reasoningcited by 1
  5. 2025GAPO: Robust Advantage Estimation for Real-World Code LLMscited by 0
  6. 2026Rethinking Reinforcement fine-tuning of LLMs: A Multi-armed Bandit Learning Perspectivecited by 0
  7. 2026Learn Where Outcomes Diverge: Efficient VLA RL via Probabilistic Chunk Maskingcited by 0
  8. 2026LC-ERD: Mining Latent Logic for Self-Evolving Reasoning via Consistency-Regulated Reward Decompositioncited by 0
  9. 2026Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Modelscited by 0
  10. 2026Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoning
  2. 2025Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting
  3. 2026When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO
  4. 2026TreeAdv: Tree-Structured Advantage Redistribution for Group-Based RL
  5. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
  6. 2026Selective-Advantage Entropy-Adaptive Horizon GRPO: Asymmetric Token-Level Discounting for Efficient Reinforcement Learning of Language Models
  7. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity
  8. 2025Evaluating GRPO and DPO for Faithful Chain-of-Thought Reasoning in LLMs