mlsanityData to 7 Oct 2026

Scheduling Your LLM Reinforcement Learning with Reasoning Trees

arXiv 2510.24832 ↗First posted 28 Oct 2025cs.AIcited by 11

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Using Reinforcement Learning with Verifiable Rewards (RLVR) to optimize Large Language Models (LLMs) can be conceptualized as progressively editing a query's `Reasoning Tree'. This process involves exploring nodes (tokens) and dynamically modifying the model's policy at each node. When combined with data scheduling, this process yields further gains in data efficiency and accuracy. However, existing RLVR data scheduling methods typically rely on path-based metrics to rank queries, overlooking the reasoning tree structures of these queries. In this paper, we introduce a novel metric, namely Reasoning Score (r-score), which measures the query's learning difficulty based on the structure of its reasoning tree. Based on the r-score, we propose the Reasoning Tree Schedule (Re-Schedule), a scheduling algorithm that constructs a curriculum progressing from structurally simple (high r-score) to complex (low r-score) queries. Experiments on six math-reasoning benchmarks show that Re-Schedule significantly improves average accuracy, achieving gains of up to 3.2%. These strong results validate our approach and demonstrate that a structural understanding of the reasoning tree provides a more powerful and principled foundation for RLVR data scheduling.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 52 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

34 of 52 references.

Closest ancestors · 22

Same direction, newest first

  1. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective11 Oct
  2. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR21 Jul
  3. 2025From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization9 Jul
  4. 2025TreeRPO: Tree Relative Policy Optimization5 Jun
  5. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  6. 2025On-Policy RL with Optimal Reward Baseline29 May
  7. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  8. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
  9. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  10. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  12. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
10 more
  1. 2025FastCuRL: Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Models21 Mar
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret3 Mar
  4. 2025LIMR: Less is More for RL Scaling17 Feb
  5. 2025Process Reinforcement through Implicit Rewards3 Feb
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  8. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  9. 2024On Designing Effective RL Reward at Training Time for LLM Reasoning19 Oct
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 12

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  3. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  4. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  5. 2025VGR: Visual Grounded Reasoningcited by 70
  6. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 67
  7. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformercited by 28
  8. 2025VFlow: Discovering Optimal Agentic Workflows for Verilog Generationcited by 19
  9. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  10. 2024World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filteringcited by 14
  11. 2025Data Efficacy for Language Model Trainingcited by 8
  12. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approachcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoningcited by 7
  2. 2026Anchored Policy Optimization: Mitigating Exploration Collapse Via Support-Constrained Rectificationcited by 5
  3. 2026GraphPO: Graph-based Policy Optimization for Reasoning Modelscited by 3
  4. 2026Believe Your Model: Distribution-Guided Confidence Calibrationcited by 2
  5. 2026Rubric-Guided Process Reward for Stepwise Model Routingcited by 2
  6. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  7. 2026When Images Look Right and Retrieve Wrong: Coverage-Guided Cross-Scale Re-Indexing for Knowledge-Faithful Generative Perceptioncited by 2
  8. 2026All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMscited by 1
  9. 2026The Pitfall of Scaling Up: Uncovering and Mitigating Popularity Bias Amplification in Scaling Transformer-based Recommenderscited by 0
  10. 2026Understanding Curriculum Learning in Large Language Models via Cross-Difficulty Optimization Dynamicscited by 0
  11. 2026Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Samplingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning at the Right Pace: Adaptive Data Scheduling Improves LLM Reinforcement Learning
  2. 2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMs
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
  4. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  5. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving
  6. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning
  7. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  8. 2026SUPERNOVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions