mlsanityData to 14 Oct 2026

Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration

arXiv 2508.13755 ↗First posted 19 Aug 2025cs.LG cs.AIcited by 43

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Reward (RLVR) is a powerful method for enhancing the reasoning abilities of Large Language Models, but its full potential is limited by a lack of exploration in two key areas: Depth (the difficulty of problems) and Breadth (the number of training instances). Our analysis of the popular GRPO algorithm reveals a bias that down-weights difficult, low-accuracy problems, which are crucial for improving reasoning skills. To address this, we introduce Difficulty Adaptive Rollout Sampling (DARS), a method that re-weights difficult problems by using targeted, multi-stage rollouts. DARS increases the number of rollout outcomes for these harder problems according to our proposed re-balancing schedules and leads to consistent gains in Pass@K. We discovered that increasing rollout size alone does not improve performance and may actually impair it. In contrast, scaling the batch size to increase breadth via full-batch updates significantly boosted Pass@1 metrics. This improvement stems from higher token-level entropy, ensuring robust exploration and minimized gradient noise. We further present DARS-Breadth, a combined approach that uses DARS with a large breadth of training data. This method demonstrates simultaneous gains in both Pass@K and Pass@1, confirming that depth (adaptive exploration) and breadth (scaling iteration instances) are orthogonal and complementary dimensions for unlocking the full power of RLVR.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 42 references that are in the map. First posted on arXiv 19 Aug 2025.

Builds on

25 of 42 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling25 Jun
  2. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning24 Jun
  3. 2025Decomposing Elements of Problem Solving: What "Math" Does RL Teach?28 May
  4. 2025Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition28 May
  5. 2025Learning to Reason under Off-Policy Guidance21 Apr
  6. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  7. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  8. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  10. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
4 more
  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  3. 2023Let's Verify Step by Step31 May
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 9

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2024OpenAI o1 System Cardcited by 1,244
  8. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  9. 2025Rethinking Reflection in Pre-Trainingcited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 33.

43 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  4. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reusecited by 29
  5. 2026Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoningcited by 27
  6. 2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 21
  7. 2025Beyond the Sampled Token: Preserving Candidate Support in RLVRcited by 18
  8. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 17
  9. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  10. 2026EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RLcited by 14
  11. 2026Learn Hard Problems During RL with Reference Guided Fine-tuningcited by 12
  12. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewardscited by 10
31 more
  1. 2026V0: A Generalist Value Model for Any Policy at State Zerocited by 7
  2. 2026When Silence Is Golden: Can LLMs Learn to Abstain in Temporal QA and Beyond?cited by 7
  3. 2026Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Informationcited by 7
  4. 2025Video-STR: Reinforcing MLLMs in Video Spatio-Temporal Reasoning with Relation Graphcited by 6
  5. 2026Accordion-Thinking: Self-Regulated Step Summaries for Efficient and Readable LLM Reasoningcited by 6
  6. 2026Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Modelscited by 6
  7. 2026Efficient RLVR Training via Weighted Mutual Information Data Selectioncited by 6
  8. 2026ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learningcited by 6
  9. 2025CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal Reasoningcited by 4
  10. 2026Low-rank Optimization Trajectories Modeling for LLM RLVR Accelerationcited by 4
  11. 2026Optimizing What Policies Learn From: Recoverability-aware Rollout Intervention Learningcited by 4
  12. 2026SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMscited by 3
  13. 2026Tail-Likelihood Reinforcement Learningcited by 3
  14. 2026Minerva: Reinforcement Learning with Verifiable Rewards for Cyber Threat Intelligence LLMscited by 2
  15. 2026WebGen-R1: Incentivizing Large Language Models to Generate Functional and Aesthetic Websites with Reinforcement Learningcited by 2
  16. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  17. 2025Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifierscited by 1
  18. 2026Group Pattern Selection Optimization: Let LRMs Pick the Right Pattern for Reasoningcited by 1
  19. 2026CuSearch: Curriculum Rollout Sampling via Search Depth for Agentic RAGcited by 1
  20. 2026Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMscited by 1
  21. 2026PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVRcited by 1
  22. 2026CircuitLens: Reasoning Circuits as Data Selection Signals for Reinforcement Learning with Verifiable Rewardscited by 1
  23. 2026Rethinking Reinforcement fine-tuning of LLMs: A Multi-armed Bandit Learning Perspectivecited by 0
  24. 2026Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purificationcited by 0
  25. 2026Demystifying Design Choices of Reinforcement Fine-tuning: A Batched Contextual Bandit Learning Perspectivecited by 0
  26. 2026When 2D Tasks Meet 1D Serialization: On Serialization Friction in Structured Taskscited by 0
  27. 2026DGPO: Beyond Pairwise Preferences with Directional Consistent Groupwise Optimizationcited by 0
  28. 2026Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoningcited by 0
  29. 2026SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learningcited by 0
  30. 2026Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVRcited by 0
  31. 2026Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  2. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  3. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  4. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
  5. 2025Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs
  6. 2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
  7. 2025Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding
  8. 2026Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVR