# Variational Student: Learning Compact and Sparser Networks in Knowledge Distillation Framework

arXiv: https://arxiv.org/abs/1910.12061 · HTML: https://mlsanity.com/paper/1910.12061/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2019.

## Built on by (most cited first, 8 of 8)

- [A Generalized Zero-Shot Quantization of Deep Convolutional Neural Networks via Learned Weights Statistics](https://mlsanity.com/paper/2112.02834/) (2021)
- [Improve Knowledge Distillation via Label Revision and Data Selection](https://mlsanity.com/paper/2404.03693/) (2024)
- [An Incremental Learning Approach to Automatically Recognize Pulmonary Diseases from the Multi-vendor Chest Radiographs](https://mlsanity.com/paper/2201.02574/) (2022)
- [Efficient Inference via Universal LSH Kernel](https://mlsanity.com/paper/2106.11426/) (2021)
- [From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks](https://mlsanity.com/paper/2405.06038/) (2024)
- [Learning Knowledge Representation with Meta Knowledge Distillation for Single Image Super-Resolution](https://mlsanity.com/paper/2207.08356/) (2022)
- [Class-aware Information for Logit-based Knowledge Distillation](https://mlsanity.com/paper/2211.14773/) (2022)
- [Layer-Wise Interpretation of Deep Neural Networks Using Identity Initialization](https://mlsanity.com/paper/2102.13333/) (2021)

## Abstract (arXiv)

The holy grail in deep neural network research is porting the memory- and computation-intensive network models on embedded platforms with a minimal compromise in model accuracy. To this end, we propose a novel approach, termed as Variational Student, where we reap the benefits of compressibility of the knowledge distillation (KD) framework, and sparsity inducing abilities of variational inference (VI) techniques. Essentially, we build a sparse student network, whose sparsity is induced by the variational parameters found via optimizing a loss function based on VI, leveraging the knowledge learnt by an accurate but complex pre-trained teacher network. Further, for sparsity enhancement, we also employ a Block Sparse Regularizer on a concatenated tensor of teacher and student network weights. We demonstrate that the marriage of KD and the VI techniques inherits compression properties from the KD framework, and enhances levels of sparsity from the VI approach, with minimal compromise in the model accuracy. We benchmark our results on LeNet MLP and VGGNet (CNN) and illustrate a memory footprint reduction of 64x and 213x on these MLP and CNN variants, respectively, without a need to retrain the teacher network. Furthermore, in the low data regime, we observed that our method outperforms state-of-the-art Bayesian techniques in terms of accuracy.