mlsanityData to 14 Oct 2026

SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning

arXiv 2506.19767 ↗First posted 24 Jun 2025cs.CL cs.AI cs.LGcited by 84

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large language models (LLMs) have achieved remarkable progress in reasoning tasks, yet the optimal integration of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) remains a fundamental challenge. Through comprehensive analysis of token distributions, learning dynamics, and integration mechanisms from entropy-based perspectives, we reveal key differences between these paradigms: SFT induces coarse-grained global changes to LLM policy distributions, while RL performs fine-grained selective optimizations, with entropy serving as a critical indicator of training effectiveness. Building on these observations, we propose Supervised Reinforcement Fine-Tuning (SRFT), a single-stage method that unifies both fine-tuning paradigms through entropy-aware weighting mechanisms. Our approach simultaneously applies SFT and RL to directly optimize the LLM using demonstrations and self-exploration rollouts rather than through two-stage sequential methods. Extensive experiments show that SRFT achieves 59.1% average accuracy, outperforming zero-RL methods by 9.0% on five mathematical reasoning benchmarks and 10.9% on three out-of-distribution benchmarks.

Key facts. Cited by 84 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 39 references that are in the map. First posted on arXiv 24 Jun 2025.

Builds on

33 of 39 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  3. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  4. 2025How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoning30 May
  5. 2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration23 May
  6. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuning22 May
  7. 2025TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoning21 May
  8. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMs19 May
  9. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  10. 2025Learning to Reason under Off-Policy Guidance21 Apr
  11. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replay19 Apr
  12. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
8 more
  1. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  2. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  4. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025Process Reinforcement through Implicit Rewards3 Feb
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 13

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  9. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  11. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  12. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 212
1 more
  1. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 32; 2026: 52.

84 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  3. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Explorationcited by 43
  4. 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36
  5. 2026Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Levelcited by 35
  6. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 31
  7. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoningcited by 30
  8. 2025Nudging the Boundaries of LLM Reasoningcited by 26
  9. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMscited by 17
  10. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 17
  11. 2025Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasonerscited by 16
  12. 2025CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systemscited by 15
38 more
  1. 2025Don't Tell the Answer, Truly Guide the Reasoning During RL Rolloutscited by 14
  2. 2025ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learningcited by 12
  3. 2026Learn Hard Problems During RL with Reference Guided Fine-tuningcited by 12
  4. 2025Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoningcited by 11
  5. 2026Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVRcited by 11
  6. 2025MTIR-SQL: Multi-turn Tool-Integrated Reasoning Reinforcement Learning for Text-to-SQLcited by 10
  7. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 10
  8. 2025Perception-Consistency Multimodal Large Language Models Reasoning via Caption-Regularized Policy Optimizationcited by 9
  9. 2026Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Modelscited by 9
  10. 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoningcited by 9
  11. 2025EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learningcited by 7
  12. 2025AWPO: Enhancing Tool-Use of Large Language Models through Adaptive Integration of Reasoning Rewardscited by 7
  13. 2026LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Modelscited by 7
  14. 2026Generative Recommendation for Large-Scale Advertisingcited by 7
  15. 2026π-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Datacited by 7
  16. 2026Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix Itcited by 7
  17. 2025Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Followingcited by 6
  18. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failurescited by 6
  19. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 6
  20. 2025Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Modelscited by 5
  21. 2025Tailored Primitive Initialization is the Secret Key to Reinforcement Learningcited by 4
  22. 2026Learning Agentic Policy from Action Guidancecited by 4
  23. 2026EchoRL: Reinforcement Learning via Rollout Echoingcited by 4
  24. 2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoffcited by 4
  25. 2025Post-Completion Learning for Language Modelscited by 3
  26. 2025Recall-Extend Dynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integrationcited by 3
  27. 2025More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Explorationcited by 3
  28. 2025Think Outside the Policy: In-Context Steered Policy Optimizationcited by 3
  29. 2025Trust-Region Adaptive Policy Optimizationcited by 3
  30. 2026PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Reviewcited by 3
  31. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  32. 2026WavAlign: Enhancing Intelligence and Expressiveness in Spoken Dialogue Models via Adaptive Hybrid Post-Trainingcited by 3
  33. 2026Learning from the Near Future: Temporal Self-Distillation for RLVRcited by 3
  34. 2026Perceive-to-Reason: Decoupling Perception and Reasoning for Fine-Grained Visual Reasoningcited by 3
  35. 2025Reasoning Pattern Matters: Learning to Reason without Human Rationalescited by 2
  36. 2025Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generationcited by 2
  37. 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoningcited by 2
  38. 2026Why Does RL Generalize Better Than SFT? A Data-Centric Perspective on VLM Post-Trainingcited by 2

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning
  2. 2025Anchored Supervised Fine-Tuning
  3. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
  4. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
  5. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy
  6. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them
  7. 2026On-Policy Supervised Fine-Tuning for Efficient Reasoning
  8. 2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training