mlsanityData to 14 Oct 2026

UFT: Unifying Supervised and Reinforcement Fine-Tuning

arXiv 2505.16984 ↗First posted 22 May 2025cs.LG cs.CLcited by 59

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Post-training has demonstrated its importance in enhancing the reasoning capabilities of large language models (LLMs). The primary post-training methods can be categorized into supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). SFT is efficient and well-suited for small language models, but it may lead to overfitting and limit the reasoning abilities of larger models. In contrast, RFT generally yields better generalization but depends heavily on the strength of the base model. To address the limitations of SFT and RFT, we propose Unified Fine-Tuning (UFT), a novel post-training paradigm that unifies SFT and RFT into a single, integrated process. UFT enables the model to effectively explore solutions while incorporating informative supervision signals, bridging the gap between memorizing and thinking underlying existing methods. Notably, UFT outperforms both SFT and RFT in general, regardless of model sizes. Furthermore, we theoretically prove that UFT breaks RFT's inherent exponential sample complexity bottleneck, showing for the first time that unified training can exponentially accelerate convergence on long-horizon reasoning tasks.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 43 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

32 of 43 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  2. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learning7 Apr
  3. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  6. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  8. 2024Free Process Rewards without Process Labels2 Dec
  9. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  10. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  11. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning8 Feb
  12. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
1 more
  1. 2023Let's Verify Step by Step31 May

Other references · 19

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2025s1: Simple test-time scalingcited by 1,341
  8. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  9. 2018Is Q-learning Provably Efficient?cited by 487
  10. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  11. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  12. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
7 more
  1. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  2. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  3. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  4. 2020Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisitedcited by 106
  5. 2022The Power of Regularization in Solving Extensive-Form Gamescited by 21
  6. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actionscited by 15
  7. 2024A Policy-Gradient Approach to Solving Imperfect-Information Games with Best-Iterate Convergencecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 23; 2026: 36.

59 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  3. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  4. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 84
  5. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 60
  6. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  7. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 34
  8. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 33
  9. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 31
  10. 2025Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuumcited by 18
  11. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMscited by 17
  12. 2025Proximal Supervised Fine-Tuningcited by 16
38 more
  1. 2025Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasonerscited by 16
  2. 2025ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learningcited by 12
  3. 2026Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoningcited by 11
  4. 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoningcited by 9
  5. 2026KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidancecited by 8
  6. 2026LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Modelscited by 7
  7. 2026Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix Itcited by 7
  8. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failurescited by 6
  9. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 6
  10. 2026ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoningcited by 6
  11. 2025AMFT: Aligning LLM Reasoners by Meta-Learning the Optimal Imitation-Exploration Balancecited by 5
  12. 2026From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoningcited by 5
  13. 2026Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Modelscited by 5
  14. 2025Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Trainingcited by 4
  15. 2026MOON3.0: Reasoning-aware Multimodal Representation Learning for E-commerce Product Understandingcited by 4
  16. 2026EchoRL: Reinforcement Learning via Rollout Echoingcited by 4
  17. 2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoffcited by 4
  18. 2025More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Explorationcited by 3
  19. 2025Think Outside the Policy: In-Context Steered Policy Optimizationcited by 3
  20. 2025Bridging VLMs and Embodied Intelligence with Deliberate Practice Policy Optimizationcited by 3
  21. 2025Aligning LLMs with Biomedical Knowledge using Balanced Fine-Tuningcited by 3
  22. 2025Trust-Region Adaptive Policy Optimizationcited by 3
  23. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  24. 2026WavAlign: Enhancing Intelligence and Expressiveness in Spoken Dialogue Models via Adaptive Hybrid Post-Trainingcited by 3
  25. 2026LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidancecited by 3
  26. 2026PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agentscited by 3
  27. 2025Reasoning Pattern Matters: Learning to Reason without Human Rationalescited by 2
  28. 2025Generalization of RLVR Using Causal Reasoning as a Testbedcited by 2
  29. 2026GIFT: Reconciling Post-Training Objectives via Variational Finite-Temperature Gibbs Initializationcited by 2
  30. 2025OraPO: Oracle-educated Reinforcement Learning for Data-efficient and Factual Radiology Report Generationcited by 1
  31. 2026KEPO: Knowledge-Enhanced Preference Optimization for Multimodal Reasoning with Applications to Medical VQAcited by 1
  32. 2026GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectificationcited by 1
  33. 2026Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regimecited by 1
  34. 2026Entropy-KL Divergence-based Token Masking: A Novel Approach for Selective Fine-tuning of Large Language Modelscited by 1
  35. 2026Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoningcited by 1
  36. 2026Think2Go: Generative Next POI Recommendation with LLM Reasoningcited by 1
  37. 2026RePO: Bridging On-Policy Learning and Off-Policy Knowledge through Rephrasing Policy Optimizationcited by 0
  38. 2026Hindsight-Anchored Policy Optimization: Learning Through Hindsight with Thompson Sampling-Inspired Adaptive Gatingcited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning
  2. 2025Anchored Supervised Fine-Tuning
  3. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMs
  4. 2025HEFT: A Coarse-to-Fine Hierarchy for Enhancing the Efficiency and Accuracy of Language Model Reasoning
  5. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions
  6. 2025CARFT: Boosting LLM Reasoning via Contrastive Learning with Annotated Chain-of-Thought-based Reinforced Fine-Tuning
  7. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning
  8. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training