mlsanityData to 14 Oct 2026

Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVR

arXiv 2609.08650 ↗First posted 8 Sep 2026cs.LG cs.AI cs.CLcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has been central to the recent success of Large Reasoning Models. However, while RLVR significantly improves single-sample accuracy, it often fails to expand the model's intrinsic reasoning coverage (pass@k) due to limited exploration during training. To address this, we optimize the structural design of train-time rollouts to enhance pass@k. Our analysis identifies three key design principles: (1) difficulty-adaptive rollout can play an important role in expanding pass@k, beyond serving as an efficiency heuristic; (2) tree-based rollout outperforms parallel sampling in discovering correct answers; and (3) sentence-entropy-guided forking overcomes the localization phenomenon of token-level branching to maximize semantic diversity. Building on these insights, we propose DATPO (Difficulty-Adaptive Sentence-entropy-guided Tree-structured Policy Optimization). DATPO integrates difficulty-adaptive tree search with a sibling-diversity advantage term, explicitly promoting semantic diversity to expand reasoning coverage during training. Experiments on mathematical reasoning benchmarks demonstrate that DATPO outperforms baselines especially in pass@k, which directly translates to superior test-time scaling performance.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 34 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

26 of 34 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026TreeAdv: Tree-Structured Advantage Redistribution for Group-Based RL7 Jan
  2. 2025Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewards28 Oct
  3. 2025Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs5 Oct
  4. 2025Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models30 Sep
  5. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse30 Sep
  6. 2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMs22 Sep
  7. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling24 Aug
  8. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  9. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  10. 2025Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs24 May
  11. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
1 more
  1. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 13

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  6. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  7. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  8. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  9. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  10. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 80
  11. 2025Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verificationcited by 50
  12. 2020PySBD: Pragmatic Sentence Boundary Disambiguationcited by 38
1 more
  1. 2024Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examplescited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learningcited by 0
  2. 2026Branching On-Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GraphPO: Graph-based Policy Optimization for Reasoning Models
  2. 2026Reinforced Efficient Reasoning via Semantically Diverse Exploration
  3. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration
  4. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
  5. 2026BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?
  6. 2026Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR
  7. 2025Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding
  8. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning