# TIP: Token Importance in On-Policy Distillation

arXiv: https://arxiv.org/abs/2604.14084 · HTML: https://mlsanity.com/paper/2604.14084/

## Key facts

- Cited by 104 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 32 of its 39 references that are in the map.
- First posted on arXiv 15 Apr 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 30 of 32)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [Not All Samples Are Created Equal: Deep Learning with Importance Sampling](https://mlsanity.com/paper/1803.00942/) (2018)
- [Entropy-Aware On-Policy Distillation of Language Models](https://mlsanity.com/paper/2603.07079/) (2026)
- [MiniLLM: On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2306.08543/) (2023)
- [OneBit: Towards Extremely Low-bit Large Language Models](https://mlsanity.com/paper/2402.11295/) (2024)
- [CRISP: Compressed Reasoning via Iterative Self-Policy Distillation](https://mlsanity.com/paper/2603.05433/) (2026)
- [DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints](https://mlsanity.com/paper/2601.18137/) (2026)
- [Densing Law of LLMs](https://mlsanity.com/paper/2412.04315/) (2024)
- [Lion: Adversarial Distillation of Proprietary Large Language Models](https://mlsanity.com/paper/2305.12870/) (2023)
- [MiniPLM: Knowledge Distillation for Pre-Training Language Models](https://mlsanity.com/paper/2410.17215/) (2024)
- [SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMs](https://mlsanity.com/paper/2510.24021/) (2025)
- [Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation](https://mlsanity.com/paper/2603.13260/) (2026)
- [Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruning](https://mlsanity.com/paper/2506.07851/) (2025)
- [LLM-Oriented Token-Adaptive Knowledge Distillation](https://mlsanity.com/paper/2510.11615/) (2025)
- [SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularization](https://mlsanity.com/paper/2511.17938/) (2025)
- [EDIS: Diagnosing LLM Reasoning via Entropy Dynamics](https://mlsanity.com/paper/2602.01288/) (2026)
- [Rethinking Selective Knowledge Distillation](https://mlsanity.com/paper/2602.01395/) (2026)
- [Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning](https://mlsanity.com/paper/2509.23873/) (2025)
- [Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning](https://mlsanity.com/paper/2602.21420/) (2026)

## Built on by (most cited first, 30 of 104)

- [Self-Distilled Agentic Reinforcement Learning](https://mlsanity.com/paper/2605.15155/) (2026)
- [Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation](https://mlsanity.com/paper/2605.26844/) (2026)
- [Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation](https://mlsanity.com/paper/2606.02684/) (2026)
- [On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents](https://mlsanity.com/paper/2606.15912/) (2026)
- [SOD: Step-wise On-policy Distillation for Small Language Model Agents](https://mlsanity.com/paper/2605.07725/) (2026)
- [Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation](https://mlsanity.com/paper/2605.11739/) (2026)
- [STEPS: Selective On-Policy Self-Distillation for Reasoning](https://mlsanity.com/paper/2605.10194/) (2026)
- [Trajectory-Refined Distillation](https://mlsanity.com/paper/2606.08432/) (2026)
- [Rubric-based On-policy Distillation](https://mlsanity.com/paper/2605.07396/) (2026)
- [SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation](https://mlsanity.com/paper/2605.07711/) (2026)
- [DOPD: Dual On-policy Distillation](https://mlsanity.com/paper/2606.30626/) (2026)
- [AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning](https://mlsanity.com/paper/2608.05987/) (2026)
- [Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training](https://mlsanity.com/paper/2605.12483/) (2026)
- [ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Tasks](https://mlsanity.com/paper/2606.27814/) (2026)
- [Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance](https://mlsanity.com/paper/2608.00782/) (2026)
- [Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation](https://mlsanity.com/paper/2608.01953/) (2026)
- [Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement](https://mlsanity.com/paper/2608.31046/) (2026)
- [Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts](https://mlsanity.com/paper/2606.21994/) (2026)
- [Mismatch Matters: On-Policy Distillation Beyond Token Agreement](https://mlsanity.com/paper/2608.09836/) (2026)
- [PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation](https://mlsanity.com/paper/2606.17199/) (2026)
- [Trust-Region Behavior Blending for On-Policy Distillation](https://mlsanity.com/paper/2605.31159/) (2026)
- [Finding the Evidence: Discovering Decision-Supporting Tokens for On-Policy Reasoning Distillation](https://mlsanity.com/paper/2606.22830/) (2026)
- [SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation](https://mlsanity.com/paper/2608.04419/) (2026)
- [A Formula-Driven Survey and Research Agenda for On-Policy Distillation](https://mlsanity.com/paper/2606.22793/) (2026)
- [TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning](https://mlsanity.com/paper/2606.32017/) (2026)
- [Group-Reflective Self-Distillation for Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.28076/) (2026)
- [Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation](https://mlsanity.com/paper/2607.10805/) (2026)
- [When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation](https://mlsanity.com/paper/2608.03632/) (2026)
- [CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillation](https://mlsanity.com/paper/2608.13387/) (2026)
- [Extremely Sparse Supervision Incentivizes Reasoning Ability](https://mlsanity.com/paper/2609.04565/) (2026)

## Abstract (arXiv)

On-policy knowledge distillation (OPD) trains a student on its own rollouts under token-level supervision from a teacher. Not all token positions matter equally, but existing views of token importance are incomplete. We ask a direct question: which tokens carry the most useful learning signal in OPD? Our answer is that informative tokens come from two regions: positions with high student entropy, and positions with low student entropy plus high teacher--student divergence, where the student is overconfident and wrong. Empirically, student entropy is a strong first-order proxy: retaining 50% of tokens with entropy-based sampling matches or exceeds all-token training while reducing peak memory by up to 47%. But entropy alone misses a second important region. When we isolate low-entropy, high-divergence tokens, training on fewer than 10% of all tokens nearly matches full-token baselines, showing that overconfident tokens carry dense corrective signal despite being nearly invisible to entropy-only rules. We organize these findings with TIP (Token Importance in on-Policy distillation), a two-axis taxonomy over student entropy and teacher--student divergence, and give a theoretical explanation for why entropy is useful yet structurally incomplete. This view motivates type-aware token selection rules that combine uncertainty and disagreement. We validate this picture across three teacher--student pairs spanning Qwen3, Llama, and Qwen2.5 on MATH-500 and AIME 2024/2025, and on the DeepPlanning benchmark for long-horizon agentic planning, where Q3-only training on <20% of tokens surpasses full-token OPD. Our experiments are implemented by extending the OPD repository https://github.com/HJSang/OPSDOnPolicyDistillation, which supports memory-efficient distillation of larger models under limited GPU budgets.