mlsanityData to 14 Oct 2026

Inductive Feedback for Mixed-Policy Distillation

arXiv 2609.35390 ↗First posted 28 Sep 2026cs.LGcited by 0

On the map: Knowledge distillation → On-policy and self-distillation for LLMs

Abstract (arXiv)

Verbal feedback can identify errors and prescribe corrections, providing rich supervision for language-model post-training even when reliable programmatic verifiers are unavailable. Such feedback, often generated by a capable model, can be used to condition the teacher in on-policy distillation, which trains the student to match the teacher's predictions on student-generated rollouts. However, this approach can transfer teacher preferences that the feedback did not motivate, while leaving much of the feedback's guidance unused. We find that both problems come from the standard on-policy distillation objective, specifically the divergence it minimizes and the distribution it uses as its target. Our proposed method addresses both limitations. First, to isolate the information conveyed by the feedback from the teacher's inherent preferences, we treat verbal feedback as evidence for or against the hypothesis that a particular token comes next at a given prefix. We then adopt a probabilistic confirmation framework which uniquely determines an ordering over the vocabulary based on the teacher's predictions before and after it receives feedback. Using a confirmation score consistent with this ordering, we construct a target distribution within a trust region of the student. Second, to learn from guidance that student rollouts can leave unused, we derive a simple shared-rollout estimator of a symmetric divergence between the student and target distributions over rollouts, reusing student and feedback-conditioned teacher rollouts in both directions through importance weighting. Empirical evaluations show that our method outperforms the common on-policy distillation recipe and a recent contrastive variant on knowledge-based and agentic benchmarks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 31 of its 39 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

31 of 39 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026Calibrating Teacher--Student Discrepancy for On-Policy Distillation18 Sep
  2. 2026On Repulsive and Attractive Teachers: Separating Correctness from Behavior in Self-Distillation18 Sep
  3. 2026On the Position Bias of On-Policy Distillation21 Jun
  4. 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents14 Jun
  5. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation10 Jun
  6. 2026EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation22 May
  7. 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning12 May
  8. 2026From Generic Correlation to Input-Specific Credit in On-Policy Self Distillation12 May
  9. 2026TIP: Token Importance in On-Policy Distillation15 Apr
  10. 2026Privileged Information Distillation for Language Models4 Feb
  11. 2026Reinforcement Learning via Self-Distillation28 Jan
  12. 2026Self-Distillation Enables Continual Learning27 Jan
1 more
  1. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan

Other references · 18

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  11. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
6 more
  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  2. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  3. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  4. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 84
  5. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  6. 2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
  2. 2026A Survey of On-Policy Distillation for Large Language Models
  3. 2026Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation
  4. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
  5. 2026Divergence-Routed On-Policy Distillation for Language Models
  6. 2026Trajectory-Refined Distillation
  7. 2026When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation
  8. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes