# Rethinking Selective Knowledge Distillation

arXiv: https://arxiv.org/abs/2602.01395 · HTML: https://mlsanity.com/paper/2602.01395/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 1 Feb 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)
- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 17 of 17)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [Selective Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2105.12967/) (2021)
- [Revisiting Knowledge Distillation for Autoregressive Language Models](https://mlsanity.com/paper/2402.11890/) (2024)
- [Keypoint-based Progressive Chain-of-Thought Distillation for LLMs](https://mlsanity.com/paper/2405.16064/) (2024)
- [SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMs](https://mlsanity.com/paper/2510.24021/) (2025)
- [Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs](https://mlsanity.com/paper/2503.16870/) (2025)
- [LLM-Oriented Token-Adaptive Knowledge Distillation](https://mlsanity.com/paper/2510.11615/) (2025)
- [EA-KD: Entropy-based Adaptive Knowledge Distillation](https://mlsanity.com/paper/2311.13621/) (2023)
- [Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation](https://mlsanity.com/paper/2512.21002/) (2025)

## Built on by (most cited first, 9 of 9)

- [TIP: Token Importance in On-Policy Distillation](https://mlsanity.com/paper/2604.14084/) (2026)
- [Less is More: Early Stopping Rollout for On-Policy Distillation](https://mlsanity.com/paper/2605.27028/) (2026)
- [SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision](https://mlsanity.com/paper/2606.28562/) (2026)
- [TISD: On-Policy Self-Distillation with Trajectory Intervention](https://mlsanity.com/paper/2609.30878/) (2026)
- [ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Models](https://mlsanity.com/paper/2609.03355/) (2026)
- [A Shared Learning Rate Is Not a Neutral Control in Selective On-Policy Distillation](https://mlsanity.com/paper/2609.22109/) (2026)
- [Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD](https://mlsanity.com/paper/2609.29142/) (2026)
- [Evidence-Aligned Multimodal On-Policy Self-Distillation for Fine-Grained Visual Understanding](https://mlsanity.com/paper/2609.34672/) (2026)
- [Learning What to Distill: Bilevel Top-K Token Selection for Self-Distillation in Large Language Models](https://mlsanity.com/paper/2610.07247/) (2026)

## Abstract (arXiv)

Growing efforts to improve knowledge distillation (KD) in large language models (LLMs) replace dense teacher supervision with selective distillation, which uses a subset of token positions, vocabulary classes, or training samples for supervision. However, it remains unclear which importance signals, selection policies, and their interplay are most effective. In this work, we revisit where and how to distill in autoregressive LLMs. We disentangle selective KD along the position, class, and sample axes and systematically compare importance signals and selection policies. Then, guided by this analysis, we identify underexplored opportunities and introduce student-entropy-guided position selection (SE-KD). Across a suite of benchmarks, SE-KD often improves accuracy, downstream task adherence, and memory efficiency over dense distillation. Extending this approach across the class and sample axes (SE-KD 3X) yields complementary efficiency gains that make offline teacher caching feasible. In practice, this reduces wall time by 70% and peak memory by 18%, while cutting storage usage by 80% over prior methods without sacrificing performance.