mlsanityData to 14 Oct 2026

Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling

arXiv 2507.01679 ↗First posted 2 Jul 2025cs.LG cs.AI cs.CLcited by 60

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Existing LLMs-post-training techniques are broadly categorized into supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). Each paradigm presents a distinct trade-off: (1) SFT excels at mimicking demonstration data, but can lead to problematic generalization as a form of behavior cloning. (2) Conversely, RFT can significantly enhance a model's performance but is prone to learning unexpected behaviors, and its performance is sensitive to the initial policy. In this paper, we propose a unified view of these methods and introduce Prefix-RFT, a hybrid approach that synergizes learning from both demonstration and exploration. Using mathematical reasoning problems as a test bed, we empirically demonstrate that Prefix-RFT is simple yet effective. Not only does it surpass the performance of standalone SFT and RFT, but it also outperforms parallel mixed-policy RFT methods. Our analysis highlights the complementary nature of SFT and RFT, validating that Prefix-RFT effectively harmonizes them. Further ablation studies confirm the method's robustness to variations in the quality and quantity of demonstration data.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 53 references that are in the map. First posted on arXiv 2 Jul 2025.

Builds on

43 of 53 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling25 Jun
  2. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective17 Jun
  3. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  4. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  5. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  6. 2025How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoning30 May
  7. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuning22 May
  8. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMs19 May
  9. 2025Learning to Reason under Off-Policy Guidance21 Apr
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  11. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  12. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
6 more
  1. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  2. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret3 Mar
  3. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  4. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  5. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training28 Jan
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 25

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  11. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  12. 2023Instruction Tuning with GPT-4cited by 686
13 more
  1. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  2. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  3. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 212
  4. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  5. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 130
  6. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 119
  7. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  8. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 45
  9. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  10. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  11. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutionscited by 26
  12. 2025A Controllable Examination for Long-Context Language Modelscited by 7
  13. 2025Active Advantage-Aligned Online Reinforcement Learning with Offline Datacited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 43.

60 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  3. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoningcited by 52
  4. 2026Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Levelcited by 35
  5. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 33
  6. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 31
  7. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMscited by 17
  8. 2025Inpainting-Guided Policy Optimization for Diffusion Large Language Modelscited by 16
  9. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  10. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Themcited by 15
  11. 2025Large Reasoning Models Learn Better Alignment from Flawed Thinkingcited by 13
  12. 2025SABER: Switchable and Balanced Training for Efficient LLM Reasoningcited by 12
38 more
  1. 2025ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learningcited by 12
  2. 2025Your Models Have Thought Enough: Training Large Reasoning Models to Stop Overthinkingcited by 10
  3. 2026Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Modelscited by 9
  4. 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoningcited by 9
  5. 2026KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidancecited by 8
  6. 2025Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMscited by 7
  7. 2026Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Informationcited by 7
  8. 2026SE-Bench: Benchmarking Self-Evolution with Knowledge Internalizationcited by 6
  9. 2026Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Modelscited by 5
  10. 2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoffcited by 4
  11. 2025Recall-Extend Dynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integrationcited by 3
  12. 2025Think Outside the Policy: In-Context Steered Policy Optimizationcited by 3
  13. 2025Toward Honest Language Models for Deductive Reasoningcited by 3
  14. 2025Trust-Region Adaptive Policy Optimizationcited by 3
  15. 2026Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidancecited by 3
  16. 2026DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoningcited by 3
  17. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  18. 2026Learning from the Near Future: Temporal Self-Distillation for RLVRcited by 3
  19. 2026Learning to Think from Multiple Thinkerscited by 3
  20. 2026The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Creditscited by 3
  21. 2026Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradientscited by 3
  22. 2026Max Out GRPO Signal: Adaptive Trace Prefix Control for Hard Reasoning Problemscited by 3
  23. 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoningcited by 2
  24. 2026TailSFT: Filtered Fine-Tuning Improves Post-Training Performancecited by 2
  25. 2026Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMscited by 1
  26. 2026KEPO: Knowledge-Enhanced Preference Optimization for Multimodal Reasoning with Applications to Medical VQAcited by 1
  27. 2026EasyVideoR1: Easier RL for Video Understandingcited by 1
  28. 2026Context Training with Active Information Seekingcited by 1
  29. 2026RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM trainingcited by 1
  30. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensionscited by 1
  31. 2026From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuationcited by 1
  32. 2026Think2Go: Generative Next POI Recommendation with LLM Reasoningcited by 1
  33. 2026Difficulty-Aware Semantic-ID Optimization for Generative Recommendationcited by 1
  34. 2026Agent-G2: Gaussian Guidance for Agentic Reinforcement Learningcited by 1
  35. 2026How Does Prefix Matter in Reasoning Model Tuning?cited by 0
  36. 2026RePO: Bridging On-Policy Learning and Off-Policy Knowledge through Rephrasing Policy Optimizationcited by 0
  37. 2026Hindsight-Anchored Policy Optimization: Learning Through Hindsight with Thompson Sampling-Inspired Adaptive Gatingcited by 0
  38. 2026Mitigating Distribution Sharpening in Math RLVR via Distribution-Aligned Hint Synthesis and Backward Hint Annealingcited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)
  2. 2026Can RL Improve Generalization of LLM Agents? An Empirical Study
  3. 2025Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasoners
  4. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
  5. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
  6. 2025Beyond Imitation: Recovering Dense Rewards from Demonstrations
  7. 2026PriFT: Prior-Support Guided Supervised Fine-Tuning
  8. 2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training