Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation
On the map: Knowledge distillation → On-policy and self-distillation for LLMs
Abstract (arXiv)
Sampled-token on-policy distillation (OPD) efficiently transfers capabilities from teacher to student using student-generated tokens, requiring teacher probabilities only for sampled tokens. Yet it frequently suffers from diversity distillation failure: the student's pass@1 improves while its pass@k plateaus, failing to inherit the teacher's diversity. To explain this, we introduce First-Order Local Entropy Influence, a signed first-order proxy that decouples each update's entropy effect into the teacher--student log-probability gap and the student's local probability structure, and empirically links entropy contraction to negative-influence positions. Motivated by this, we propose Influence-Directed Adaptive On-Policy Distillation (IDA-OPD): rather than relying on costly full-vocabulary Forward-KL objectives, it preserves entropy-expanding updates while replacing entropy-contracting ones with divergence-adaptive advantage shrinkage, using only the teacher's sampled-token log-probability. Experiments on reasoning-oriented distillation show IDA-OPD consistently improves pass@k, inheriting the teacher's diversity through distillation, matches the strongest teacher-informed methods at strictly lower cost, and broadly maintains vanilla OPD's pass@1, all without full-vocabulary teacher information.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 19 of its 22 references that are in the map. First posted on arXiv 30 Aug 2026.
Builds on
19 of 22 references.
Closest ancestors · 10
Same direction, newest first
- 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation1 Jun
- 2026Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation26 May
- 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation8 May
- 2026Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level7 May
- 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization6 May
- 2026TIP: Token Importance in On-Policy Distillation15 Apr
- 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes26 Mar
- 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillation11 Mar
- 2026Entropy-Aware On-Policy Distillation of Language Models7 Mar
- 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation12 Feb
Other references · 9
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
- 2016Sequence-Level Knowledge Distillationcited by 801
- 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
- 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
- 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 90
- 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learningcited by 11
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Token Utility for On-Policy Distillation via Local Trust-Region Optimization
- 2026Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
- 2026DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillation
- 2026Fast and Effective On-policy Distillation from Reasoning Prefixes
- 2026Divergence-Routed On-Policy Distillation for Language Models
- 2026OPRD: On-Policy Representation Distillation
- 2026SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision
- 2026DOPD: Dual On-policy Distillation