mlsanityData to 7 Oct 2026

Rubric-based On-policy Distillation

arXiv 2605.07396 ↗First posted 8 May 2026cs.LG cs.AIcited by 11

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

On-policy distillation (OPD) is a powerful paradigm for model alignment, yet its reliance on teacher logits restricts its application to white-box scenarios. We contend that structured semantic rubrics can serve as a scalable alternative to teacher logits, enabling OPD using only teacher-generated responses. To prove it, we introduce ROPD, a simple yet foundational framework for rubric-based OPD. Specifically, ROPD induces prompt-specific rubrics from teacher-student contrasts, and then utilizes these rubrics to score the student rollouts for on-policy optimization. Empirically, ROPD outperforms the advanced logit-based OPD methods across most scenarios, and achieving up to a 10x gain in sample efficiency. These results position rubric-based OPD as a flexible, black-box-compatible alternative to the prevailing logit-based OPD, offering a simple yet strong baseline for scalable distillation across proprietary and open-source LLMs. Code is available at https://github.com/Peregrine123/ROPD_official.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 33 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

32 of 33 references.

References · 32

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  7. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  8. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  9. 2016Sequence-Level Knowledge Distillationcited by 761
  10. 2015Policy Distillationcited by 414
  11. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  12. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
20 more
  1. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  2. 2026MiMo-V2-Flash Technical Reportcited by 163
  3. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  4. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  5. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 148
  6. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  7. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  8. 2025Reinforcement Learning with Rubric Anchorscited by 108
  9. 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 98
  10. 2026TIP: Token Importance in On-Policy Distillationcited by 90
  11. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 53
  12. 2026Fast and Effective On-policy Distillation from Reasoning Prefixescited by 43
  13. 2026SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weightingcited by 38
  14. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 28
  15. 2026PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competencecited by 26
  16. 2026OVD: On-policy Verbal Distillationcited by 9
  17. 2026SibylSense: Adaptive Rubric Learning via Memory Tuning and Adversarial Probingcited by 6
  18. 2025A Dual-Space Framework for General Knowledge Distillation of Large Language Modelscited by 5
  19. 2025ORPO-Distill: Mixed-Policy Preference Optimization for Cross-Architecture LLM Distillationcited by 4
  20. 2026Reinforcement Learning-based Semi-supervised Knowledge Distillation with LLM-as-a-Judgecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026DanceOPD: On-Policy Generative Field Distillationcited by 12
  2. 2026Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Trainingcited by 9
  3. 2026Rethinking Reward Supervision: Rubric-Conditioned Self-Distillationcited by 9
  4. 2026TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learningcited by 5
  5. 2026LOTAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoningcited by 4
  6. 2026Who Teaches Which Token? Verifier-Gated Multi-Expert On-Policy Distillation for Scientific Reasoningcited by 2
  7. 2026CataOPD: Catalytic On-Policy Distillation for Large Language Model Reasoningcited by 0
  8. 2026Persistent Negatives for Adversarial Black-Box On-Policy Distillationcited by 0
  9. 2026UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agentscited by 0
  10. 2026Rubric-Aware On-Policy Self-Distillation for LLM Personalizationcited by 0
  11. 2026Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shiftscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe
  2. 2026When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation
  3. 2026Simple-OPD: Demystifying Warm-up for On-policy Distillation
  4. 2026OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation
  5. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
  6. 2026Towards Understanding On-Policy Distillation through the Lens of Test-Time Scaling
  7. 2026What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection
  8. 2026Contrastive On-Policy Distillation