# On the Geometry of On-Policy Distillation

arXiv: https://arxiv.org/abs/2606.07082 · HTML: https://mlsanity.com/paper/2606.07082/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 5 Jun 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 13 of 13)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes](https://mlsanity.com/paper/2603.25562/) (2026)
- [The Invisible Leash: Why RLVR May or May Not Escape Its Origin](https://mlsanity.com/paper/2507.14843/) (2025)
- [The Path Not Taken: RLVR Provably Learns Off the Principals](https://mlsanity.com/paper/2511.08567/) (2025)
- [Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation](https://mlsanity.com/paper/2605.11739/) (2026)
- [LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning](https://mlsanity.com/paper/2506.00772/) (2025)

## Built on by (most cited first, 19 of 19)

- [Rethinking On-Policy Distillation of Large Language Models II: One Training Example](https://mlsanity.com/paper/2609.04172/) (2026)
- [SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning](https://mlsanity.com/paper/2608.14277/) (2026)
- [RISE: Recursive Improvement via Self-Extrapolating Policy Distillation](https://mlsanity.com/paper/2609.05295/) (2026)
- [Geometric Self-Distillation for Reasoning Generalization](https://mlsanity.com/paper/2607.06855/) (2026)
- [DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment](https://mlsanity.com/paper/2607.07820/) (2026)
- [SCOPE-OPSD: Fisher-Conditioned Privileged Subspaces for On-Policy Self-Distillation](https://mlsanity.com/paper/2609.12579/) (2026)
- [PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning](https://mlsanity.com/paper/2609.36642/) (2026)
- [GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs](https://mlsanity.com/paper/2608.11674/) (2026)
- [Normalized Low-Rank Adaptation](https://mlsanity.com/paper/2608.31036/) (2026)
- [Train What You Deploy: Closing the MLP Reachability Gap in Low-Rank Clone Distillation](https://mlsanity.com/paper/2609.02006/) (2026)
- [Understanding the Synergy between SFT, RLVR, and OPD in LLM Post-Training](https://mlsanity.com/paper/2609.31900/) (2026)
- [USA: Update-aware SAM for Cross-domain On-Policy Disitllation of Language Agents](https://mlsanity.com/paper/2609.34225/) (2026)
- [PMOPD: Task Ordering, Cycling, and Parameter-Update Subspace Protection in Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2609.34605/) (2026)
- [On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Training](https://mlsanity.com/paper/2609.36659/) (2026)
- [No Task Vector Is an Island: A Comprehensive Study on the Composability of Task Vectors from On-Policy Distillation](https://mlsanity.com/paper/2609.39405/) (2026)
- [From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2610.02179/) (2026)
- [UP-MOPD: Update Projection in Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2610.08398/) (2026)
- [Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals](https://mlsanity.com/paper/2610.11247/) (2026)
- [Geometric Calibration of Supervised Fine-Tuning](https://mlsanity.com/paper/2610.18634/) (2026)

## Abstract (arXiv)

On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood. We characterize the trajectory of OPD updates in parameter space and compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). A suite of parameter-space diagnostics consistently places OPD in a relaxed off-principal regime: compared with SFT, its updates affect fewer weights and avoid principal directions more strongly, while compared with RLVR, they remain less tightly constrained. Beyond this static localization, OPD exhibits subspace locking: its cumulative updates rapidly enter a narrow low-dimensional channel. Constraining training to the update subspace formed early in training preserves OPD performance but substantially degrades SFT, indicating that the locked subspace is functionally sufficient for OPD. Control experiments further show that sparsifying the update tokens and shifting rollout generation off-policy preserve the rank dynamics, whereas mixing the OPD objective with RLVR changes them. Overall, these results suggest that OPD is not merely an intermediate point between SFT and RLVR, but induces its own update geometry in parameter space.