mlsanityData to 14 Oct 2026

Self-Distillation with Verbal Regularization

arXiv 2610.16354 ↗First posted 12 Oct 2026cs.CLcited by 0

On the map: Knowledge distillation → On-policy and self-distillation for LLMs

Abstract (arXiv)

On-policy self-distillation (OPSD) trains a large language model using supervision from a teacher instantiated from the same model as the student. The teacher is conditioned on privileged information (PI), such as verified solutions. The student, which does not observe PI, learns from the teacher's next-token distributions along its own sampled responses. However, PI can bias supervision toward shortcut reasoning with limited verification or reconsideration, accompanied by a sharp drop in the entropy of the output distribution. To address this challenge, we introduce self-distillation with verbal regularization (VRSD), which uses system prompts to regulate how the teacher uses PI, thereby encouraging rethinking and exploration to counteract premature commitment. Theoretically, we model prompting as attenuating the influence of PI on the teacher distribution and derive conditions for entropy preservation. Across three models and four tasks, VRSD improves Pass@1 over OPSD in 11 of 12 model-task settings, by 3.9 percentage points on average, while generally maintaining higher token entropy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 28 of its 31 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

28 of 31 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation10 Aug
  2. 2026On-Policy Self-Distillation without Any Supervision6 Aug
  3. 2026Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation5 Aug
  4. 2026Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoning3 Aug
  5. 2026Rethinking On-Policy Self-Distillation for Thinking Models6 Jul
  6. 2026Purified OPSD: On-Policy Self-Distillation Without Losing How to Think2 Jul
  7. 2026HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation10 Jun
  8. 2026Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation12 May
  9. 2026Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information12 May
  10. 2026Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning12 May
  11. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents12 Apr
  12. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?25 Mar
3 more
  1. 2026On-Policy Context Distillation for Language Models12 Feb
  2. 2026Reinforcement Learning via Self-Distillation28 Jan
  3. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan

Other references · 13

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  5. 2020Knowledge Distillation: A Surveycited by 1,603
  6. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  7. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  8. 2016Sequence-Level Knowledge Distillationcited by 801
  9. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  10. 2022Large Language Models Are Reasoning Teacherscited by 329
  11. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  12. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 90
1 more
  1. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 58

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
  2. 2026Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models
  3. 2026Recursive Self-Improvement via On-Policy Distillation for Reasoning
  4. 2026DemoPSD: Disagreement-Modulated Policy Self-Distillation
  5. 2026Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation
  6. 2026Rethinking Privileged Information in On-Policy Self-Distillation
  7. 2026One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context
  8. 2026Diagnosing On-Policy Self-Distillation for Reasoning Language Models