# Towards On-Policy SFT: Distribution Discriminant Theory and its Applications in LLM Training

arXiv: https://arxiv.org/abs/2602.12222 · HTML: https://mlsanity.com/paper/2602.12222/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 47 references that are in the map.
- First posted on arXiv 12 Feb 2026.

## Builds on (most cited first, 30 of 35)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [LeanDojo: Theorem Proving with Retrieval-Augmented Language Models](https://mlsanity.com/paper/2306.15626/) (2023)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation](https://mlsanity.com/paper/2401.08417/) (2024)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning](https://mlsanity.com/paper/2504.11456/) (2025)
- [Self-Distillation Enables Continual Learning](https://mlsanity.com/paper/2601.19897/) (2026)
- [MathScale: Scaling Instruction Tuning for Mathematical Reasoning](https://mlsanity.com/paper/2403.02884/) (2024)
- [On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification](https://mlsanity.com/paper/2508.05629/) (2025)
- [Cumulative Reasoning with Large Language Models](https://mlsanity.com/paper/2308.04371/) (2023)
- [Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts](https://mlsanity.com/paper/2402.10958/) (2024)
- [ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selection](https://mlsanity.com/paper/2601.09195/) (2026)

## Built on by (most cited first, 11 of 11)

- [A Survey of On-Policy Distillation for Large Language Models](https://mlsanity.com/paper/2604.00626/) (2026)
- [Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation](https://mlsanity.com/paper/2606.02684/) (2026)
- [Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability](https://mlsanity.com/paper/2604.06628/) (2026)
- [Learning from Your Own Mistakes: Constructing Learnable Micro-Reflective Trajectories for Self-Distillation](https://mlsanity.com/paper/2606.18844/) (2026)
- [PriFT: Prior-Support Guided Supervised Fine-Tuning](https://mlsanity.com/paper/2606.09396/) (2026)
- [ProteinOPD: Towards Effective and Efficient Preference Alignment for Protein Design](https://mlsanity.com/paper/2605.10189/) (2026)
- [Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2608.26735/) (2026)
- [On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics](https://mlsanity.com/paper/2609.35259/) (2026)
- [PG-SFT: Balancing Capability Acquisition and Retention in Offline Agent Fine-Tuning](https://mlsanity.com/paper/2610.00949/) (2026)
- [Learning What to Imitate: Entropy-Aware Distribution Mixing](https://mlsanity.com/paper/2610.06671/) (2026)
- [What Should SFT Learn? Conditional Information Gain for Generalizable Post-Training](https://mlsanity.com/paper/2610.12995/) (2026)

## Abstract (arXiv)

Supervised fine-tuning (SFT) is computationally efficient but often yields inferior generalization compared to reinforcement learning (RL). This gap is primarily driven by RL's use of on-policy data. We propose a framework to bridge this chasm by enabling On-Policy SFT. We first present textbftextitDistribution Discriminant Theory (DDT), which explains and quantifies the alignment between data and the model-induced distribution. Leveraging DDT, we introduce two complementary techniques: (i) textbftextitIn-Distribution Finetuning (IDFT), a loss-level method to enhance generalization ability of SFT, and (ii) textbftextitHinted Decoding, a data-level technique that can re-align the training corpus to the model's distribution. Extensive experiments demonstrate that our framework achieves generalization performance surpassing prominent offline RL algorithms, including DPO and SimPO, while maintaining the efficiency of an SFT pipeline. The proposed framework thus offers a practical alternative in domains where RL is infeasible. We open-source the code here: https://github.com/zhangmiaosen2000/Towards-On-Policy-SFT