mlsanityData to 14 Oct 2026

On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification

arXiv 2508.05629 ↗First posted 7 Aug 2025cs.LGcited by 127

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

In this work, we present a simple yet theoretically motivated improvement to Supervised Fine-Tuning (SFT) for the Large Language Model (LLM), addressing its limited generalization compared to reinforcement learning (RL). Through mathematical analysis, we reveal that standard SFT gradients implicitly encode a problematic reward structure that may severely restrict the generalization capabilities of model compared to RL. To rectify this, we propose Dynamic Fine-Tuning (model), stabilizing gradient updates for each token by dynamically rescaling the objective function with the probability of this token. With just a single-line change, the method outperforms standard SFT on multiple difficult benchmarks and base models, from math reasoning to code generation and multi-modal tasks, demonstrating improved generalization. Additionally, model~achieves competitive results in offline RL settings, providing an effective yet streamlined alternative. By bridging theoretical insights with practical solutions, this work advances the state of SFT. The source code will be available at https://github.com/yongliang-wu/DFT.

Key facts. Cited by 127 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 69 of its 93 references that are in the map. First posted on arXiv 7 Aug 2025.

Builds on

69 of 93 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)17 Jul
  2. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning3 Mar
  3. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training28 Jan

Other references · 66

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
38 more
  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  4. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  6. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  7. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2025Group Sequence Policy Optimizationcited by 835
  10. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  11. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  12. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  13. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  14. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  15. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
  16. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  17. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  18. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 355
  19. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 286
  20. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 284
  21. 2025Learning to Reason under Off-Policy Guidancecited by 281
  22. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  23. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 128
  24. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 106
  25. 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 99
  26. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Modelscited by 97
  27. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  28. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  29. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  30. 2020Text Generation by Learning from Demonstrationscited by 64
  31. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuningcited by 59
  32. 2019A Mean Field Theory of Batch Normalizationcited by 58
  33. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  34. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  35. 2025SPEED: Scalable, Precise, and Efficient Concept Erasure for Diffusion Modelscited by 36
  36. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustmentcited by 30
  37. 2024Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correctingcited by 26
  38. 2025Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimizationcited by 22

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 43; 2026: 84.

127 papers.

Sort:

By area:

  1. 2026Self-Distillation Enables Continual Learningcited by 247
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  4. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  5. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 59
  6. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 59
  7. 2025Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?cited by 34
  8. 2025On Predictability of Reinforcement Learning Dynamics for Large Language Modelscited by 23
  9. 2025Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuningcited by 20
  10. 2025Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuumcited by 18
  11. 2026ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selectioncited by 18
  12. 2025Proximal Supervised Fine-Tuningcited by 16
38 more
  1. 2025Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasonerscited by 16
  2. 2026Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgettingcited by 16
  3. 2025Anchored Supervised Fine-Tuningcited by 15
  4. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friendscited by 15
  5. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMscited by 15
  6. 2026Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignmentcited by 13
  7. 2025SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularizationcited by 12
  8. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 11
  9. 2025VideoNSA: Native Sparse Attention Scales Video Understandingcited by 11
  10. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 11
  11. 2026Towards On-Policy SFT: Distribution Discriminant Theory and its Applications in LLM Trainingcited by 11
  12. 2025Executable Counterfactuals: Improving LLMs' Causal Reasoning Through Codecited by 10
  13. 2025Real-Time Motion-Controllable Autoregressive Video Diffusioncited by 9
  14. 2026Probability-Entropy Calibration: An Elastic Indicator for Adaptive Fine-tuningcited by 9
  15. 2026Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Modelscited by 9
  16. 2025Variational Reasoning for Language Modelscited by 8
  17. 2025One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradientcited by 8
  18. 2025Debunk the Myth of SFT Generalizationcited by 8
  19. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  20. 2025Tiny Model, Big Logic: Diversity-Driven Optimization Elicits Large-Model Reasoning Ability in VibeThinker-1.5Bcited by 8
  21. 2026ATPO: Adaptive Tree Policy Optimization for Multi-Turn Medical Dialoguecited by 8
  22. 2026RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMscited by 8
  23. 2026Distilled Reinforcement Learning for LLM Post-trainingcited by 8
  24. 2025RFG: Test-Time Scaling for Diffusion Large Language Model Reasoning with Reward-Free Guidancecited by 7
  25. 2026SED-SFT: Selectively Encouraging Diversity in Supervised Fine-Tuningcited by 7
  26. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigationcited by 7
  27. 2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weightcited by 6
  28. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 6
  29. 2026On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Lengthcited by 6
  30. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?cited by 5
  31. 2025Metis-SPECS: Decoupling Multimodal Learning via Self-distilled Preference-based Cold Startcited by 5
  32. 2025The End of Manual Decoding: Towards Truly End-to-End Language Modelscited by 5
  33. 2026FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptationcited by 5
  34. 2025GIFT: Guided Importance-Aware Fine-Tuning for Diffusion Language Modelscited by 4
  35. 2026On-Policy Supervised Fine-Tuning for Efficient Reasoningcited by 4
  36. 2026Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Trainingcited by 4
  37. 2026InfoSFT: Learn More and Forget Less with Information-Aware Token Weightingcited by 4
  38. 2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoffcited by 4

50 most cited of 127 All 127 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
  2. 2024Unlock the Correlation between Supervised Fine-Tuning and Reinforcement Learning in Training Code Large Language Models
  3. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them
  4. 2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training
  5. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning
  6. 2025Reinforcement learning fine-tuning of language model for instruction following and math reasoning
  7. 2026TMS: Trajectory-Mixed Supervision for Reward-Free, On-Policy SFT
  8. 2026SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs