# Distillation ≈ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network

arXiv: https://arxiv.org/abs/1910.01255 · HTML: https://mlsanity.com/paper/1910.01255/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2019.

## Built on by (most cited first, 28 of 28)

- [Self-Improvement in Language Models: The Sharpening Mechanism](https://mlsanity.com/paper/2412.01951/) (2024)
- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification](https://mlsanity.com/paper/2406.07515/) (2024)
- [Selective Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2105.12967/) (2021)
- [Rethinking Influence Functions of Neural Networks in the Over-parameterized Regime](https://mlsanity.com/paper/2112.08297/) (2021)
- [Revisiting Self-Distillation](https://mlsanity.com/paper/2206.08491/) (2022)
- [Understanding Self-Distillation in the Presence of Label Noise](https://mlsanity.com/paper/2301.13304/) (2023)
- [Class-Distribution-Aware Calibration for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2109.05263/) (2021)
- [On the benefits of knowledge distillation for adversarial robustness](https://mlsanity.com/paper/2203.07159/) (2022)
- [Respecting Transfer Gap in Knowledge Distillation](https://mlsanity.com/paper/2210.12787/) (2022)
- [Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning](https://mlsanity.com/paper/2101.08732/) (2021)
- [Churn Reduction via Distillation](https://mlsanity.com/paper/2106.02654/) (2021)
- [Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher](https://mlsanity.com/paper/2110.08532/) (2021)
- [On student-teacher deviations in distillation: does it pay to disobey?](https://mlsanity.com/paper/2301.12923/) (2023)
- [Knowledge Distillation: Bad Models Can Be Good Role Models](https://mlsanity.com/paper/2203.14649/) (2022)
- [Resurrecting Label Propagation for Graphs with Heterophily and Label Noise](https://mlsanity.com/paper/2310.16560/) (2023)
- [Distilling Double Descent](https://mlsanity.com/paper/2102.06849/) (2021)
- [Self-paced Resistance Learning against Overfitting on Noisy Labels](https://mlsanity.com/paper/2105.03059/) (2021)
- [Retraining with Predicted Hard Labels Provably Increases Model Accuracy](https://mlsanity.com/paper/2406.11206/) (2024)
- [Knowledge Distillation ≈ Label Smoothing: Fact or Fallacy?](https://mlsanity.com/paper/2301.12609/) (2023)
- [Knowledge Distillation with Noisy Labels for Natural Language Understanding](https://mlsanity.com/paper/2109.10147/) (2021)
- [Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing](https://mlsanity.com/paper/2505.15195/) (2025)
- [Revisiting Knowledge Distillation: The Hidden Role of Dataset Size](https://mlsanity.com/paper/2510.15516/) (2025)
- [Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models](https://mlsanity.com/paper/2605.17778/) (2026)
- [Mixed-Type Wafer Classification For Low Memory Devices Using Knowledge Distillation](https://mlsanity.com/paper/2303.13974/) (2023)
- [Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation](https://mlsanity.com/paper/2504.14307/) (2025)
- [SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelines](https://mlsanity.com/paper/2601.01484/) (2026)
- [Plug, Play, and Fortify: A Low-Cost Module for Robust Multimodal Image Understanding Models](https://mlsanity.com/paper/2602.22644/) (2026)

## Abstract (arXiv)

Distillation is a method to transfer knowledge from one model to another and often achieves higher accuracy with the same capacity. In this paper, we aim to provide a theoretical understanding on what mainly helps with the distillation. Our answer is "early stopping". Assuming that the teacher network is overparameterized, we argue that the teacher network is essentially harvesting dark knowledge from the data via early stopping. This can be justified by a new concept, Anisotropic Information Retrieval (AIR), which means that the neural network tends to fit the informative information first and the non-informative information (including noise) later. Motivated by the recent development on theoretically analyzing overparameterized neural networks, we can characterize AIR by the eigenspace of the Neural Tangent Kernel(NTK). AIR facilities a new understanding of distillation. With that, we further utilize distillation to refine noisy labels. We propose a self-distillation algorithm to sequentially distill knowledge from the network in the previous training epoch to avoid memorizing the wrong labels. We also demonstrate, both theoretically and empirically, that self-distillation can benefit from more than just early stopping. Theoretically, we prove convergence of the proposed algorithm to the ground truth labels for randomly initialized overparameterized neural networks in terms of ℓ2 distance, while the previous result was on convergence in 0-1 loss. The theoretical result ensures the learned neural network enjoy a margin on the training data which leads to better generalization. Empirically, we achieve better testing accuracy and entirely avoid early stopping which makes the algorithm more user-friendly.