mlsanityData to 14 Oct 2026

Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMs

arXiv 2505.13026 ↗First posted 19 May 2025cs.LG cs.AIcited by 25

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large language models (LLMs) excel at mathematical reasoning and logical problem-solving. The current popular training paradigms primarily use supervised fine-tuning (SFT) and reinforcement learning (RL) to enhance the models' reasoning abilities. However, when using SFT or RL alone, there are respective challenges: SFT may suffer from overfitting, while RL is prone to mode collapse. The state-of-the-art methods have proposed hybrid training schemes. However, static switching faces challenges such as poor generalization across different tasks and high dependence on data quality. In response to these challenges, inspired by the curriculum learning-quiz mechanism in human reasoning cultivation, We propose SASR, a step-wise adaptive hybrid training framework that theoretically unifies SFT and RL and dynamically balances the two throughout optimization. SASR uses SFT for initial warm-up to establish basic reasoning skills, and then uses an adaptive dynamic adjustment algorithm based on gradient norm and divergence relative to the original distribution to seamlessly integrate SFT with the online RL method GRPO. By monitoring the training status of LLMs and adjusting the training process in sequence, SASR ensures a smooth transition between training schemes, maintaining core reasoning abilities while exploring different paths. Experimental results demonstrate that SASR outperforms SFT, RL, and static hybrid training methods.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 37 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

28 of 37 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  6. 2024Teaching Large Language Models to Reason with Reinforcement Learning7 Mar
  7. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 20

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  10. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  11. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 391
  12. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 284
8 more
  1. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  2. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  3. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  4. 2024Language Models Learn to Mislead Humans via RLHFcited by 124
  5. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 97
  6. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  7. 2024Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reductioncited by 18
  8. 2024Investigating Mysteries of CoT-Augmented Distillationcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 13.

25 papers.

Sort:

By area:

  1. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  2. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 84
  3. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 60
  4. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimizationcited by 35
  5. 2026Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Levelcited by 35
  6. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 34
  7. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoningcited by 30
  8. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 17
  9. 2025Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasonerscited by 16
  10. 2025CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systemscited by 15
  11. 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoningcited by 9
  12. 2025EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learningcited by 7
13 more
  1. 2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoffcited by 4
  2. 2025Recall-Extend Dynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integrationcited by 3
  3. 2025Rethinking Large Language Model Distillation: A Constrained Markov Decision Process Perspectivecited by 3
  4. 2025Toward Honest Language Models for Deductive Reasoningcited by 3
  5. 2026Gradients Must Earn Their Influence: Unifying SFT with Generalized Entropic Objectivescited by 3
  6. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  7. 2026GIFT: Reconciling Post-Training Objectives via Variational Finite-Temperature Gibbs Initializationcited by 2
  8. 2026Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answeringcited by 2
  9. 2026KEPO: Knowledge-Enhanced Preference Optimization for Multimodal Reasoning with Applications to Medical VQAcited by 1
  10. 2026AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolutioncited by 1
  11. 2026Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidancecited by 0
  12. 2026MERGED: Multimodal Entity Resolution via Generated Expert Reasoning Distillationcited by 0
  13. 2026The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions
  2. 2026Application-Driven Pedagogical Knowledge Optimization of Open-Source LLMs via Reinforcement Learning and Supervised Fine-Tuning
  3. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them
  4. 2026SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs
  5. 2026Dual-Phase LLM Reasoning: Self-Evolved Mathematical Frameworks
  6. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learning
  7. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuning
  8. 2025CARFT: Boosting LLM Reasoning via Contrastive Learning with Annotated Chain-of-Thought-based Reinforced Fine-Tuning