mlsanityData to 7 Oct 2026

OPRD: On-Policy Representation Distillation

arXiv 2606.06021 ↗First posted 4 Jun 2026cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

On-policy distillation (OPD) supervises the student exclusively in the output space by matching next-token distributions. This paradigm suffers from two limitations: (i) a high-variance gradient estimator whose signal-to-noise ratio collapses as the student approaches the teacher, and (ii) an LM-head information bottleneck that discards the teacher's intermediate hidden states. We propose On-Policy Representation Distillation (OPRD), the first method to lift on-policy distillation into the hidden-state space. OPRD aligns student and teacher representations across selected layers on the same on-policy rollouts, providing dense, deterministic, per-layer supervision while bypassing the LM head entirely. Theoretically, OPRD provides a deterministic per-sample gradient, removing the token-level estimation variance that plagues OPD, and exposes structural information that any output-space objective necessarily discards. Empirically, OPRD closes the student-teacher gap on competition mathematics benchmarks (AIME 2024, AIME 2025, and AIMO), where every output-space baseline plateaus below the teacher, while training 1.44x faster and using up to 54% less memory. We further extend OPRD to the cross-architecture setting via OPRD-Bridge. By exploiting the observation that heterogeneous models share a low-rank representational structure, we construct a frozen projector pair that aligns representations across arbitrary depth and width mismatches, shifting the alignment from the output space (which depends on a shared vocabulary) to the representation space. We validate OPRD-Bridge on both cross-architecture (Qwen3-4B -> Qwen3-1.7B-Base) and cross-tokenizer (Phi-4-mini-reasoning -> Qwen3-1.7B-Base) settings, demonstrating successful knowledge transfer even when the vocabulary-based alignment channel is unavailable. Code: https://github.com/ShenzhiYang2000/OPRD.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 47 of its 49 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

47 of 49 references.

References · 47

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  9. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  10. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
35 more
  1. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  2. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  3. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  4. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  6. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  7. 2016Sequence-Level Knowledge Distillationcited by 761
  8. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  9. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  10. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  11. 2014Deeply-Supervised Netscited by 364
  12. 2024The Platonic Representation Hypothesiscited by 362
  13. 2026Reinforcement Learning via Self-Distillationcited by 285
  14. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  15. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  16. 2026Self-Distillation Enables Continual Learningcited by 212
  17. 2022Relative representations enable zero-shot latent space communicationcited by 175
  18. 2026MiMo-V2-Flash Technical Reportcited by 163
  19. 2026On-Policy Context Distillation for Language Modelscited by 161
  20. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  21. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  22. 2022Linearly Mapping from Image to Text Spacecited by 139
  23. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  24. 2026Self-Distilled RLVRcited by 109
  25. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 87
  26. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 71
  27. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 65
  28. 2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipecited by 57
  29. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 49
  30. 2026Stable On-Policy Distillation through Adaptive Target Reformulationcited by 47
  31. 2025Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Mathcited by 44
  32. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  33. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillationcited by 39
  34. 2026How Far Can Unsupervised RLVR Scale LLM Training?cited by 26
  35. 2026Online Experiential Learning for Language Modelscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Weak-to-Strong On-Policy Distillationcited by 7
  2. 2026Mismatch Matters: On-Policy Distillation Beyond Token Agreementcited by 7
  3. 2026Cliff: Learning Process Rewards from the First Mistakecited by 1
  4. 2026Negative Self-Distillation: Learning to Reason by Avoiding Flawscited by 1
  5. 2026SCOPE-OPSD: Fisher-Conditioned Privileged Subspaces for On-Policy Self-Distillationcited by 1
  6. 2026LastOPD: Taming Collapse in Latent On-Policy Distillationcited by 1
  7. 2026Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoningcited by 0
  8. 2026The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillationcited by 0
  9. 2026PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learningcited by 0