Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
Not on the map. Why?
Abstract (arXiv)
Pre-trained Large Language Model (LLM) exhibits broad capabilities, yet, for specific tasks or domains their attainment of higher accuracy and more reliable reasoning generally depends on post-training through Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL). Although often treated as distinct methodologies, recent theoretical and empirical developments demonstrate that SFT and RL are closely connected. This study presents a comprehensive and unified perspective on LLM post-training with SFT and RL. We first provide an in-depth overview of both techniques, examining their objectives, algorithmic structures, and data requirements. We then systematically analyze their interplay, highlighting frameworks that integrate SFT and RL, hybrid training pipelines, and methods that leverage their complementary strengths. Drawing on a representative set of recent application studies from 2023 to 2025, we identify emerging trends, characterize the rapid shift toward hybrid post-training paradigms, and distill key takeaways that clarify when and why each method is most effective. By synthesizing theoretical insights, practical methodologies, and empirical evidence, this study establishes a coherent understanding of SFT and RL within a unified framework and outlines promising directions for future research in scalable, efficient, and generalizable LLM post-training.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 122 of its 147 references that are in the map. First posted on arXiv 14 Mar 2026.
Builds on
122 of 147 references.
References · 122
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
- 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
- 2023LIMA: Less Is More for Alignmentcited by 1,080
38 more
- 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
- 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
- 2022Large Language Models Can Self-Improvecited by 656
- 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
- 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
- 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
- 2023Llemma: An Open Language Model For Mathematicscited by 385
- 2020Generative Language Modeling for Automated Theorem Provingcited by 379
- 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
- 2019Causal Confusion in Imitation Learningcited by 286
- 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
- 2025Learning to Reason under Off-Policy Guidancecited by 266
- 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
- 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
- 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
- 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
- 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
- 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
- 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 155
- 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
- 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
- 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
- 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
- 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimizationcited by 110
- 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
- 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
- 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
- 2025Self-Evolving Curriculum for LLM Reasoningcited by 107
- 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 102
- 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
- 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 87
- 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 86
- 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 85
- 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learningcited by 84
- 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
- 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 80
- 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoningcited by 75
50 most cited of 122 All 122 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillationcited by 5
- 2026On Distinguishing Capability Elicitation from Capability Creation in Post-Training: A Free-Energy Perspectivecited by 0
- 2026Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidancecited by 0
- 2026CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillation13 Aug 2026
- 2026Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidance14 May 2026
- 2026On Distinguishing Capability Elicitation from Capability Creation in Post-Training: A Free-Energy Perspective8 May 2026