mlsanityData to 14 Oct 2026

Towards On-Policy SFT: Distribution Discriminant Theory and its Applications in LLM Training

arXiv 2602.12222 ↗First posted 12 Feb 2026cs.LG cs.AI cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Supervised fine-tuning (SFT) is computationally efficient but often yields inferior generalization compared to reinforcement learning (RL). This gap is primarily driven by RL's use of on-policy data. We propose a framework to bridge this chasm by enabling On-Policy SFT. We first present textbftextitDistribution Discriminant Theory (DDT), which explains and quantifies the alignment between data and the model-induced distribution. Leveraging DDT, we introduce two complementary techniques: (i) textbftextitIn-Distribution Finetuning (IDFT), a loss-level method to enhance generalization ability of SFT, and (ii) textbftextitHinted Decoding, a data-level technique that can re-align the training corpus to the model's distribution. Extensive experiments demonstrate that our framework achieves generalization performance surpassing prominent offline RL algorithms, including DPO and SimPO, while maintaining the efficiency of an SFT pipeline. The proposed framework thus offers a practical alternative in domains where RL is infeasible. We open-source the code here: https://github.com/zhangmiaosen2000/Towards-On-Policy-SFT

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 12 Feb 2026.

Builds on

35 of 47 references.

References · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2023A Survey of Large Language Modelscited by 2,576
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
23 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  2. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  4. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  5. 2025Large Language Diffusion Modelscited by 912
  6. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  7. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  8. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 427
  9. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  10. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  11. 2026Reinforcement Learning via Self-Distillationcited by 332
  12. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 280
  13. 2026Self-Distillation Enables Continual Learningcited by 247
  14. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  15. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  16. 2023Cumulative Reasoning with Large Language Modelscited by 94
  17. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Promptscited by 36
  18. 2026ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selectioncited by 18
  19. 2025Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasonerscited by 16
  20. 2026Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgettingcited by 16
  21. 2025Anchored Supervised Fine-Tuningcited by 15
  22. 2025Debunk the Myth of SFT Generalizationcited by 8
  23. 2021On Generalization of Adversarial Imitation Learning and Beyondcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  2. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 32
  3. 2026Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capabilitycited by 14
  4. 2026Learning from Your Own Mistakes: Constructing Learnable Micro-Reflective Trajectories for Self-Distillationcited by 4
  5. 2026PriFT: Prior-Support Guided Supervised Fine-Tuningcited by 2
  6. 2026ProteinOPD: Towards Effective and Efficient Preference Alignment for Protein Designcited by 0
  7. 2026Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillationcited by 0
  8. 2026On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamicscited by 0
  9. 2026PG-SFT: Balancing Capability Acquisition and Retention in Offline Agent Fine-Tuningcited by 0
  10. 2026Learning What to Imitate: Entropy-Aware Distribution Mixingcited by 0
  11. 2026What Should SFT Learn? Conditional Information Gain for Generalizable Post-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Finetuning with Sampling: SFT Learns Better Than You Think
  2. 2025Proximal Supervised Fine-Tuning
  3. 2026Post-Training is About States, Not Tokens: A State Distribution View of SFT, RL, and On-Policy Distillation
  4. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
  5. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
  6. 2025DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Training
  7. 2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)
  8. 2025One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradient