mlsanityData to 14 Oct 2026

Proximal Supervised Fine-Tuning

arXiv 2508.17784 ↗First posted 25 Aug 2025cs.LG cs.AI cs.CLcited by 16

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Supervised fine-tuning (SFT) of foundation models often leads to poor generalization, where prior capabilities deteriorate after tuning on new tasks or domains. Inspired by trust-region policy optimization (TRPO) and proximal policy optimization (PPO) in reinforcement learning (RL), we propose Proximal SFT (PSFT). This fine-tuning objective incorporates the benefits of trust-region, effectively constraining policy drift during SFT while maintaining competitive tuning. By viewing SFT as a special case of policy gradient methods with constant positive advantages, we derive PSFT that stabilizes optimization and leads to generalization, while leaving room for further optimization in subsequent post-training stages. Experiments across mathematical and human-value domains show that PSFT matches SFT in-domain, outperforms it in out-of-domain generalization, remains stable under prolonged training without causing entropy collapse, and provides a stronger foundation for the subsequent optimization.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 52 of its 60 references that are in the map. First posted on arXiv 25 Aug 2025.

Builds on

52 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting15 Aug
  2. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification7 Aug
  3. 2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)17 Jul
  4. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training28 Jan

Other references · 48

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2025Qwen2.5-VL Technical Reportcited by 5,587
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
36 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  10. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2025s1: Simple test-time scalingcited by 1,341
  13. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  14. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  15. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  16. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  17. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  18. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  19. 2025LIMO: Less is More for Reasoningcited by 498
  20. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  21. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  22. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  23. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
  24. 2025Learning to Reason under Off-Policy Guidancecited by 281
  25. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  26. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 106
  27. 2019HEAD-QA: A Healthcare Dataset for Complex Reasoningcited by 101
  28. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 70
  29. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuningcited by 59
  30. 2025Reasoning to Learn from Latent Thoughtscited by 45
  31. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  32. 2025NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Taskscited by 32
  33. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 23
  34. 2024Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Modelcited by 22
  35. 2025Does Learning Mathematical Problem-Solving Generalize to Broader Reasoning?cited by 7
  36. 2025Flexible Realignment of Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 11.

16 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuumcited by 18
  3. 2026Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Modelscited by 9
  4. 2026Hybrid Policy Distillation for LLMscited by 9
  5. 2025Debunk the Myth of SFT Generalizationcited by 8
  6. 2026Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RLcited by 8
  7. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 6
  8. 2026On-Policy Supervised Fine-Tuning for Efficient Reasoningcited by 4
  9. 2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoffcited by 4
  10. 2025Toward Honest Language Models for Deductive Reasoningcited by 3
  11. 2025EtCon: Edit-then-Consolidate for Reliable Knowledge Editingcited by 1
  12. 2026Fine-Tuning Without Forgetting via Loss-Adaptive Learning Ratescited by 1
4 more
  1. 2026Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoningcited by 1
  2. 2026ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Trainingcited by 0
  3. 2026Selecting Diverse SFT Traces Improves Post-RL Generalizationcited by 0
  4. 2026Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PriFT: Prior-Support Guided Supervised Fine-Tuning
  2. 2025Trust-Region Adaptive Policy Optimization
  3. 2026Towards On-Policy SFT: Distribution Discriminant Theory and its Applications in LLM Training
  4. 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning
  5. 2025Polychromic Objectives for Reinforcement Learning
  6. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
  7. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
  8. 2025Anchored Supervised Fine-Tuning