# SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning

arXiv: https://arxiv.org/abs/2506.19767 · HTML: https://mlsanity.com/paper/2506.19767/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 33 of its 39 references that are in the map.
- First posted on arXiv 24 Jun 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 30 of 33)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [Process Reinforcement through Implicit Rewards](https://mlsanity.com/paper/2502.01456/) (2025)
- [Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model](https://mlsanity.com/paper/2503.24290/) (2025)
- [Reinforcement Learning for Reasoning in Large Language Models with One Training Example](https://mlsanity.com/paper/2504.20571/) (2025)
- [VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks](https://mlsanity.com/paper/2504.05118/) (2025)
- [Learning to Reason under Off-Policy Guidance](https://mlsanity.com/paper/2504.14945/) (2025)
- [SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models](https://mlsanity.com/paper/2504.11468/) (2025)
- [ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models](https://mlsanity.com/paper/2505.24864/) (2025)
- [Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions](https://mlsanity.com/paper/2506.07527/) (2025)
- [UFT: Unifying Supervised and Reinforcement Fine-Tuning](https://mlsanity.com/paper/2505.16984/) (2025)
- [LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities](https://mlsanity.com/paper/2504.16078/) (2025)
- [Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration](https://mlsanity.com/paper/2505.17621/) (2025)
- [TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2505.15692/) (2025)

## Built on by (most cited first, 30 of 84)

- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting](https://mlsanity.com/paper/2508.11408/) (2025)
- [Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration](https://mlsanity.com/paper/2508.13755/) (2025)
- [Towards a Unified View of Large Language Model Post-Training](https://mlsanity.com/paper/2509.04419/) (2025)
- [Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level](https://mlsanity.com/paper/2605.06387/) (2026)
- [RL Fine-Tuning Heals OOD Forgetting in SFT](https://mlsanity.com/paper/2509.12235/) (2025)
- [Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning](https://mlsanity.com/paper/2509.06948/) (2025)
- [Nudging the Boundaries of LLM Reasoning](https://mlsanity.com/paper/2509.25666/) (2025)
- [RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMs](https://mlsanity.com/paper/2508.16546/) (2025)
- [Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2602.01601/) (2026)
- [Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasoners](https://mlsanity.com/paper/2510.04454/) (2025)
- [CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems](https://mlsanity.com/paper/2509.24088/) (2025)
- [Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts](https://mlsanity.com/paper/2510.09388/) (2025)
- [ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning](https://mlsanity.com/paper/2512.13095/) (2025)
- [Learn Hard Problems During RL with Reference Guided Fine-tuning](https://mlsanity.com/paper/2603.01223/) (2026)
- [Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning](https://mlsanity.com/paper/2508.09883/) (2025)
- [Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR](https://mlsanity.com/paper/2609.04108/) (2026)
- [MTIR-SQL: Multi-turn Tool-Integrated Reasoning Reinforcement Learning for Text-to-SQL](https://mlsanity.com/paper/2510.25510/) (2025)
- [Multi-Rollout On-Policy Distillation via Peer Successes and Failures](https://mlsanity.com/paper/2605.12652/) (2026)
- [Perception-Consistency Multimodal Large Language Models Reasoning via Caption-Regularized Policy Optimization](https://mlsanity.com/paper/2509.21854/) (2025)
- [Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models](https://mlsanity.com/paper/2602.02244/) (2026)
- [SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning](https://mlsanity.com/paper/2604.23747/) (2026)
- [EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning](https://mlsanity.com/paper/2508.07809/) (2025)
- [AWPO: Enhancing Tool-Use of Large Language Models through Adaptive Integration of Reasoning Rewards](https://mlsanity.com/paper/2512.19126/) (2025)
- [LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Models](https://mlsanity.com/paper/2602.14147/) (2026)
- [Generative Recommendation for Large-Scale Advertising](https://mlsanity.com/paper/2602.22732/) (2026)
- [π-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data](https://mlsanity.com/paper/2604.14054/) (2026)
- [Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It](https://mlsanity.com/paper/2606.26027/) (2026)
- [Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Following](https://mlsanity.com/paper/2508.03178/) (2025)
- [Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failures](https://mlsanity.com/paper/2601.19928/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have achieved remarkable progress in reasoning tasks, yet the optimal integration of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) remains a fundamental challenge. Through comprehensive analysis of token distributions, learning dynamics, and integration mechanisms from entropy-based perspectives, we reveal key differences between these paradigms: SFT induces coarse-grained global changes to LLM policy distributions, while RL performs fine-grained selective optimizations, with entropy serving as a critical indicator of training effectiveness. Building on these observations, we propose Supervised Reinforcement Fine-Tuning (SRFT), a single-stage method that unifies both fine-tuning paradigms through entropy-aware weighting mechanisms. Our approach simultaneously applies SFT and RL to directly optimize the LLM using demonstrations and self-exploration rollouts rather than through two-stage sequential methods. Extensive experiments show that SRFT achieves 59.1% average accuracy, outperforming zero-RL methods by 9.0% on five mathematical reasoning benchmarks and 10.9% on three out-of-distribution benchmarks.