# Knowledge Distillation with Feature Maps for Image Classification

arXiv: https://arxiv.org/abs/1812.00660 · HTML: https://mlsanity.com/paper/1812.00660/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2018.

## Built on by (most cited first, 17 of 17)

- [Knowledge Distillation as Semiparametric Inference](https://mlsanity.com/paper/2104.09732/) (2021)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillation](https://mlsanity.com/paper/2112.03473/) (2021)
- [Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation](https://mlsanity.com/paper/2407.03056/) (2024)
- [Knowledge Distillation in Vision Transformers: A Critical Review](https://mlsanity.com/paper/2302.02108/) (2023)
- [Resolution-Based Distillation for Efficient Histology Image Classification](https://mlsanity.com/paper/2101.04170/) (2021)
- [Spectral Co-Distillation for Personalized Federated Learning](https://mlsanity.com/paper/2401.17124/) (2024)
- [U-Know-DiffPAN: An Uncertainty-aware Knowledge Distillation Diffusion Framework with Details Enhancement for PAN-Sharpening](https://mlsanity.com/paper/2412.06243/) (2024)
- [Mitigating Semantic Collapse in Partially Relevant Video Retrieval](https://mlsanity.com/paper/2510.27432/) (2025)
- [What is Left After Distillation? How Knowledge Transfer Impacts Fairness and Bias](https://mlsanity.com/paper/2410.08407/) (2024)
- [A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene](https://mlsanity.com/paper/2404.11249/) (2024)
- [FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learning](https://mlsanity.com/paper/2406.02355/) (2024)
- [AME: Aligned Manifold Entropy for Robust Vision-Language Distillation](https://mlsanity.com/paper/2508.08644/) (2025)
- [Scheduled Knowledge Acquisition on Lightweight Vector Symbolic Architectures for Brain-Computer Interfaces](https://mlsanity.com/paper/2403.13844/) (2024)
- [Leveraging Different Learning Styles for Improved Knowledge Distillation in Biomedical Imaging](https://mlsanity.com/paper/2212.02931/) (2022)
- [Model compression using knowledge distillation with integrated gradients](https://mlsanity.com/paper/2506.14440/) (2025)
- [A Novel Compression Framework for YOLOv8: Achieving Real-Time Aerial Object Detection on Edge Devices via Structured Pruning and Channel-Wise Distillation](https://mlsanity.com/paper/2509.12918/) (2025)

## Abstract (arXiv)

The model reduction problem that eases the computation costs and latency of complex deep learning architectures has received an increasing number of investigations owing to its importance in model deployment. One promising method is knowledge distillation (KD), which creates a fast-to-execute student model to mimic a large teacher network. In this paper, we propose a method, called KDFM (Knowledge Distillation with Feature Maps), which improves the effectiveness of KD by learning the feature maps from the teacher network. Two major techniques used in KDFM are shared classifier and generative adversarial network. Experimental results show that KDFM can use a four layers CNN to mimic DenseNet-40 and use MobileNet to mimic DenseNet-100. Both student networks have less than 1% accuracy loss comparing to their teacher models for CIFAR-100 datasets. The student networks are 2-6 times faster than their teacher models for inference, and the model size of MobileNet is less than half of DenseNet-100's.