mlsanityData to 7 Oct 2026

BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning

arXiv 2506.17211 ↗First posted 20 Jun 2025cs.LGcited by 57

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Small language models (SLMs) struggle to learn complex reasoning behaviors, especially when high-quality traces are scarce or difficult to learn from. The standard training approach combines a supervised fine-tuning (SFT) stage, often to distill capabilities of a larger model, followed by a reinforcement learning (RL)stage such as Group Relative Policy Optimization (GRPO). In this paper, we investigate the fundamental limitations of this SFT + RL paradigm and propose methods to overcome them. Under a suitable theoretical model, we demonstrate that the SFT + RL strategy can fail completely when (1) the expert's traces are too difficult for the small model to express, or (2) the small model's initialization has exponentially small likelihood of success. To address these, we introduce BREAD: a GRPO variant that unifies the SFT and RL stages via partial expert guidance and branched rollouts. When self-generated traces fail, BREAD adaptively inserts short expert prefixes/hints, allowing the small model to complete the rest of the reasoning path, and ensuring that each update includes at least one successful trace. This mechanism both densifies the reward signal and induces a natural learning curriculum. BREAD requires fewer than 40% of ground-truth traces, consistently outperforming standard GRPO while speeding up the training by about 3 times. Importantly, we demonstrate that BREAD helps the model solve problems that are otherwise unsolvable by the SFT + RL strategy, highlighting how branched rollouts and expert guidance can substantially boost SLM reasoning.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 44 references that are in the map. First posted on arXiv 20 Jun 2025.

Builds on

26 of 44 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  2. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  3. 2025DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training13 Apr
  4. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning10 Mar
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan

Other references · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  7. 2025s1: Simple test-time scalingcited by 1,311
  8. 2024OpenAI o1 System Cardcited by 1,220
  9. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  10. 2024Phi-4 Technical Reportcited by 675
  11. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  12. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
6 more
  1. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  2. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 223
  3. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155
  4. 2025Small Models Struggle to Learn from Strong Reasonerscited by 109
  5. 2018Kickstarting Deep Reinforcement Learningcited by 75
  6. 2025Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcementcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 44.

57 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
  3. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  4. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoningcited by 48
  5. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 39
  6. 2025StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reasoncited by 37
  7. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMscited by 17
  8. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  9. 2026InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoningcited by 14
  10. 2025Don't Tell the Answer, Truly Guide the Reasoning During RL Rolloutscited by 13
  11. 2026Learn Hard Problems During RL with Reference Guided Fine-tuningcited by 11
  12. 2025ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learningcited by 10
38 more
  1. 2026Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignmentcited by 10
  2. 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoningcited by 8
  3. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
  4. 2025Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMscited by 7
  5. 2026KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidancecited by 7
  6. 2026Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Informationcited by 6
  7. 2026Measuring Weak-to-Strong Legibility of Reasoning Modelscited by 5
  8. 2026Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVRcited by 4
  9. 2025Recall-Extend Dynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integrationcited by 3
  10. 2025Think Outside the Policy: In-Context Steered Policy Optimizationcited by 3
  11. 2026Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioningcited by 3
  12. 2026Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidancecited by 3
  13. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  14. 2026ICRL: Learning to Internalize Self-Critique with Reinforcement Learningcited by 3
  15. 2026From Off-Policy to On-Policy: Enhancing GUI Agents via Bi-level Expert-to-Policy Assimilationcited by 2
  16. 2026RPO:Reinforcement Fine-Tuning with Partial Reasoning Optimizationcited by 2
  17. 2026P^2O: Joint Policy and Prompt Optimizationcited by 2
  18. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimizationcited by 2
  19. 2026VISTA: View-Consistent Self-Verified Training for GUI Groundingcited by 2
  20. 2025AAPA: Adversarially Anchored Preference Alignment for Post-Training of Large Language Modelscited by 1
  21. 2026Making Expert Reasoning Learnable with Self-Distillationcited by 1
  22. 2026VAM: Verbalized Action Masking for Controllable Exploration in RL Post-Training -- A Chess Case Studycited by 1
  23. 2025SmartChunk Retrieval: Query-Aware Chunk Compression with Planning for Efficient Document RAGcited by 1
  24. 2026Decouple before Integration: Test-time Synthesis of SFT and RLVR Task Vectorscited by 1
  25. 2026Self-Trained Verification for Training- and Test-Time Self-Improvementcited by 1
  26. 2026Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoningcited by 1
  27. 2026It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branchescited by 1
  28. 2026Difficulty-Aware Semantic-ID Optimization for Generative Recommendationcited by 1
  29. 2025CORE: Concept-Oriented Reinforcement for Bridging the Definition-Application Gap in Mathematical Reasoningcited by 0
  30. 2026KEPO: Knowledge-Enhanced Preference Optimization for Multimodal Reasoning with Applications to Medical VQAcited by 0
  31. 2026RePO: Bridging On-Policy Learning and Off-Policy Knowledge through Rephrasing Policy Optimizationcited by 0
  32. 2026Context Bootstrapped Reinforcement Learningcited by 0
  33. 2026Mitigating Distribution Sharpening in Math RLVR via Distribution-Aligned Hint Synthesis and Backward Hint Annealingcited by 0
  34. 2026Memory as a Markov Matrix: Sample Efficient Knowledge Expansion via Token-to-Dictionary Mappingcited by 0
  35. 2026VSPO: Vector-Steered Policy Optimization for Behavioral Controlcited by 0
  36. 2026POLARIS: Guiding Small Models to Write Long Storiescited by 0
  37. 2026The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithmscited by 0
  38. 2026Reasoning to Regulate: Chain-of-Thought for Traffic Rule Understandingcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning
  2. 2025G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance
  3. 2026GRPO Training Dynamics for Small Language Models
  4. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration
  5. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  6. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning
  7. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
  8. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback