# On-Policy Self-Distillation without Any Supervision

arXiv: https://arxiv.org/abs/2608.06296 · HTML: https://mlsanity.com/paper/2608.06296/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 56 of its 59 references that are in the map.
- First posted on arXiv 6 Aug 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 30 of 56)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [LIMO: Less is More for Reasoning](https://mlsanity.com/paper/2502.03387/) (2025)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Scaling Relationship on Learning Mathematical Reasoning with Large Language Models](https://mlsanity.com/paper/2308.01825/) (2023)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [OpenThoughts: Data Recipes for Reasoning Models](https://mlsanity.com/paper/2506.04178/) (2025)
- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention](https://mlsanity.com/paper/2506.13585/) (2025)
- [Self-Distillation Enables Continual Learning](https://mlsanity.com/paper/2601.19897/) (2026)
- [R-Zero: Self-Evolving Reasoning LLM from Zero Data](https://mlsanity.com/paper/2508.05004/) (2025)
- [Learning to Reason without External Rewards](https://mlsanity.com/paper/2505.19590/) (2025)
- [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026)
- [RL's Razor: Why Online Reinforcement Learning Forgets Less](https://mlsanity.com/paper/2509.04259/) (2025)
- [Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond](https://mlsanity.com/paper/2503.10460/) (2025)

## Built on by (most cited first, 17 of 17)

- [TTPO: Test-Time Policy Optimization](https://mlsanity.com/paper/2608.27448/) (2026)
- [Negative Self-Distillation: Learning to Reason by Avoiding Flaws](https://mlsanity.com/paper/2609.11699/) (2026)
- [What Does Privileged Information Add to On-Policy Self-Distillation?](https://mlsanity.com/paper/2609.20612/) (2026)
- [What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillation](https://mlsanity.com/paper/2609.25623/) (2026)
- [When Clipping Reverses Correction: Failure Dynamics of Pointwise Forward-KL On-Policy Self-Distillation](https://mlsanity.com/paper/2609.38995/) (2026)
- [Learn How to Act from Your Own Interactions: On-Policy Self-Distillation for GUI Agents](https://mlsanity.com/paper/2609.27307/) (2026)
- [Thinking Leakage: A Causal Audit of NoThink Post-Training in Hybrid Reasoning Models](https://mlsanity.com/paper/2609.28682/) (2026)
- [Progressive-View On-Policy Distillation for Regional-to-Global Transfer in Multimodal LLMs](https://mlsanity.com/paper/2609.32333/) (2026)
- [Expected Reasoning-Step Return Unifies On-Policy Learning from Rewards and Teachers](https://mlsanity.com/paper/2609.32674/) (2026)
- [SeOPD: Self-Evolving LLMs via Online Policy Distillation from Self-Generated Chain-of-Thought](https://mlsanity.com/paper/2609.33181/) (2026)
- [d-OPD: Future-Aware On-Policy Distillation for Block Diffusion Language Models](https://mlsanity.com/paper/2609.35362/) (2026)
- [Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2609.37132/) (2026)
- [Privileged Context as Drift in On-Policy Self-Distillation](https://mlsanity.com/paper/2610.07842/) (2026)
- [Self-Distillation with Verbal Regularization](https://mlsanity.com/paper/2610.16354/) (2026)
- [PaceReasoner: Label-Free Learning to Answer Briefly from Long Reasoning](https://mlsanity.com/paper/2610.17900/) (2026)
- [Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning](https://mlsanity.com/paper/2610.19176/) (2026)
- [Teaching Agents in their Own Style: Blackbox Teacher Distillation for Web Agents](https://mlsanity.com/paper/2610.19834/) (2026)

## Abstract (arXiv)

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively. Code is available at [https://github.com/williamium3000/u-opsd](https://github.com/williamium3000/u-opsd).