# Energy-efficient Knowledge Distillation for Spiking Neural Networks arXiv: https://arxiv.org/abs/2106.07172 ยท HTML: https://mlsanity.com/paper/2106.07172/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Placed in Distillation for spiking neural networks, one of 12 directions in Knowledge distillation. - Builds on 26 of its 42 references that are in the map. - First posted on arXiv 14 Jun 2021. ## Placed in - Knowledge distillation / [Distillation for spiking neural networks](https://mlsanity.com/distill/distillation-for-spiking-neural-networks/) ## Builds on (most cited first, 26 of 26) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015) - [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015) - [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014) - [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019) - [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016) - [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017) - [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019) - [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016) - [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019) - [Going Deeper With Directly-Trained Larger Spiking Neural Networks](https://mlsanity.com/paper/2011.05280/) (2020) - [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018) - [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015) - [Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures](https://mlsanity.com/paper/1903.06379/) (2019) - [STDP Based Pruning of Connections and Weight Quantization in Spiking Neural Networks for Energy Efficient Recognition](https://mlsanity.com/paper/1710.04734/) (2017) - [T2FSNN: Deep Spiking Neural Networks with Time-to-first-spike Coding](https://mlsanity.com/paper/2003.11741/) (2020) - [Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization](https://mlsanity.com/paper/1911.00822/) (2019) - [Exploring the Connection Between Binary and Spiking Neural Networks](https://mlsanity.com/paper/2002.10064/) (2020) - [Optimizing the energy consumption of spiking neural networks for neuromorphic applications](https://mlsanity.com/paper/1912.01268/) (2019) - [Fast and Efficient Information Transmission with Burst Spikes in Deep Spiking Neural Networks](https://mlsanity.com/paper/1809.03142/) (2018) - [Event-Driven Visual-Tactile Sensing and Learning for Robots](https://mlsanity.com/paper/2009.07083/) (2020) - [Distilling Spikes: Knowledge Distillation in Spiking Neural Networks](https://mlsanity.com/paper/2005.00288/) (2020) - [Spatio-Temporal Pruning and Quantization for Low-latency Spiking Neural Networks](https://mlsanity.com/paper/2104.12528/) (2021) - [SpikeGrad: An ANN-equivalent Computation Model for Implementing Backpropagation with Spikes](https://mlsanity.com/paper/1906.00851/) (2019) - [Autonomous Quadrotor Flight despite Rotor Failure with Onboard Vision Sensors: Frames vs. Events](https://mlsanity.com/paper/2102.13406/) (2021) - [Spiking neural networks trained with backpropagation for low power neuromorphic implementation of voice activity detection](https://mlsanity.com/paper/1910.09993/) (2019) ## Built on by (most cited first, 11 of 11) - [LaSNN: Layer-wise ANN-to-SNN Distillation for Effective and Efficient Training in Deep Spiking Neural Networks](https://mlsanity.com/paper/2304.09101/) (2023) - [Biologically inspired structure learning with reverse knowledge distillation for spiking neural networks](https://mlsanity.com/paper/2304.09500/) (2023) - [BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge Distillation](https://mlsanity.com/paper/2407.09083/) (2024) - [Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networks](https://mlsanity.com/paper/2503.03144/) (2025) - [Adaptive Sparse Structure Development with Pruning and Regeneration for Spiking Neural Networks](https://mlsanity.com/paper/2211.12219/) (2022) - [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024) - [Developmental Plasticity-inspired Adaptive Pruning for Deep Spiking and Artificial Neural Networks](https://mlsanity.com/paper/2211.12714/) (2022) - [Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacement](https://mlsanity.com/paper/2503.16572/) (2025) - [CADE: Cosine Annealing Differential Evolution for Spiking Neural Network](https://mlsanity.com/paper/2406.02349/) (2024) - [Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment](https://mlsanity.com/paper/2501.15925/) (2025) - [Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training](https://mlsanity.com/paper/2510.06254/) (2025) ## Abstract (arXiv) Spiking neural networks (SNNs) have been gaining interest as energy-efficient alternatives of conventional artificial neural networks (ANNs) due to their event-driven computation. Considering the future deployment of SNN models to constrained neuromorphic devices, many studies have applied techniques originally used for ANN model compression, such as network quantization, pruning, and knowledge distillation, to SNNs. Among them, existing works on knowledge distillation reported accuracy improvements of student SNN model. However, analysis on energy efficiency, which is also an important feature of SNN, was absent. In this paper, we thoroughly analyze the performance of the distilled SNN model in terms of accuracy and energy efficiency. In the process, we observe a substantial increase in the number of spikes, leading to energy inefficiency, when using the conventional knowledge distillation methods. Based on this analysis, to achieve energy efficiency, we propose a novel knowledge distillation method with heterogeneous temperature parameters. We evaluate our method on two different datasets and show that the resulting SNN student satisfies both accuracy improvement and reduction of the number of spikes. On MNIST dataset, our proposed student SNN achieves up to 0.09% higher accuracy and produces 65% less spikes compared to the student SNN trained with conventional knowledge distillation method. We also compare the results with other SNN compression techniques and training methods.