# Iterative Self Knowledge Distillation -- From Pothole Classification to Fine-Grained and COVID Recognition

arXiv: https://arxiv.org/abs/2202.02265 · HTML: https://mlsanity.com/paper/2202.02265/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 4 Feb 2022.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations](https://mlsanity.com/paper/2106.01548/) (2021)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)
- [Knowledge Distillation Meets Self-Supervision](https://mlsanity.com/paper/2006.07114/) (2020)
- [Knowledge Distillation via Route Constrained Optimization](https://mlsanity.com/paper/1904.09149/) (2019)
- [Heterogeneous Knowledge Distillation using Information Flow Modeling](https://mlsanity.com/paper/2005.00727/) (2020)
- [Road Damage Detection Using Deep Neural Networks with Images Captured Through a Smartphone](https://mlsanity.com/paper/1801.09454/) (2018)
- [Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledge](https://mlsanity.com/paper/2004.00176/) (2020)

## Built on by (most cited first, 1 of 1)

- [Joint Training of Image Generator and Detector for Road Defect Detection](https://mlsanity.com/paper/2509.03465/) (2025)

## Abstract (arXiv)

Pothole classification has become an important task for road inspection vehicles to save drivers from potential car accidents and repair bills. Given the limited computational power and fixed number of training epochs, we propose iterative self knowledge distillation (ISKD) to train lightweight pothole classifiers. Designed to improve both the teacher and student models over time in knowledge distillation, ISKD outperforms the state-of-the-art self knowledge distillation method on three pothole classification datasets across four lightweight network architectures, which supports that self knowledge distillation should be done iteratively instead of just once. The accuracy relation between the teacher and student models shows that the student model can still benefit from a moderately trained teacher model. Implying that better teacher models generally produce better student models, our results justify the design of ISKD. In addition to pothole classification, we also demonstrate the efficacy of ISKD on six additional datasets associated with generic classification, fine-grained classification, and medical imaging application, which supports that ISKD can serve as a general-purpose performance booster without the need of a given teacher model and extra trainable parameters.