# Distilling Spikes: Knowledge Distillation in Spiking Neural Networks

arXiv: https://arxiv.org/abs/2005.00288 · HTML: https://mlsanity.com/paper/2005.00288/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 May 2020.

## Built on by (most cited first, 27 of 27)

- [Constructing Deep Spiking Neural Networks from Artificial Neural Networks with Knowledge Distillation](https://mlsanity.com/paper/2304.05627/) (2023)
- [Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorization](https://mlsanity.com/paper/2305.02099/) (2023)
- [Direct Learning-Based Deep Spiking Neural Networks: A Review](https://mlsanity.com/paper/2305.19725/) (2023)
- [Spatio-Temporal Pruning and Quantization for Low-latency Spiking Neural Networks](https://mlsanity.com/paper/2104.12528/) (2021)
- [Coaching a Teachable Student](https://mlsanity.com/paper/2306.10014/) (2023)
- [SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillation](https://mlsanity.com/paper/2308.15122/) (2023)
- [LaSNN: Layer-wise ANN-to-SNN Distillation for Effective and Efficient Training in Deep Spiking Neural Networks](https://mlsanity.com/paper/2304.09101/) (2023)
- [Energy-efficient Knowledge Distillation for Spiking Neural Networks](https://mlsanity.com/paper/2106.07172/) (2021)
- [Biologically inspired structure learning with reverse knowledge distillation for spiking neural networks](https://mlsanity.com/paper/2304.09500/) (2023)
- [BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge Distillation](https://mlsanity.com/paper/2407.09083/) (2024)
- [Temporal Knowledge Sharing enable Spiking Neural Network Learning from Past and Future](https://mlsanity.com/paper/2304.06540/) (2023)
- [Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networks](https://mlsanity.com/paper/2503.03144/) (2025)
- [Self-Distillation Learning Based on Temporal-Spatial Consistency for Spiking Neural Networks](https://mlsanity.com/paper/2406.07862/) (2024)
- [Towards Low-latency Event-based Visual Recognition with Hybrid Step-wise Distillation Spiking Neural Networks](https://mlsanity.com/paper/2409.12507/) (2024)
- [Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacement](https://mlsanity.com/paper/2503.16572/) (2025)
- [Applications of Deep Learning in Fish Habitat Monitoring: A Tutorial and Survey](https://mlsanity.com/paper/2206.05394/) (2022)
- [Cross Knowledge Distillation between Artificial and Spiking Neural Networks](https://mlsanity.com/paper/2507.09269/) (2025)
- [Razor SNN: Efficient Spiking Neural Network with Temporal Embeddings](https://mlsanity.com/paper/2306.17597/) (2023)
- [A Novel Spike Transformer Network for Depth Estimation from Event Cameras via Cross-modality Knowledge Distillation](https://mlsanity.com/paper/2404.17335/) (2024)
- [Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment](https://mlsanity.com/paper/2501.15925/) (2025)
- [Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers](https://mlsanity.com/paper/2510.07924/) (2025)
- [Head-Tail-Aware KL Divergence in Knowledge Distillation for Spiking Neural Networks](https://mlsanity.com/paper/2504.20445/) (2025)
- [Toward Efficient Deep Spiking Neuron Networks:A Survey On Compression](https://mlsanity.com/paper/2407.08744/) (2024)
- [Spiking Brain Compression: Post-Training Second-order Compression for Spiking Neural Networks](https://mlsanity.com/paper/2506.03996/) (2025)
- [Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training](https://mlsanity.com/paper/2510.06254/) (2025)
- [MD-SNN: Membrane Potential-aware Distillation on Quantized Spiking Neural Network](https://mlsanity.com/paper/2512.04443/) (2025)
- [Active Spiking Perception: The Membrane Potential as a Belief State for Anytime 3D Point Cloud Recognition](https://mlsanity.com/paper/2608.19232/) (2026)

## Abstract (arXiv)

Spiking Neural Networks (SNN) are energy-efficient computing architectures that exchange spikes for processing information, unlike classical Artificial Neural Networks (ANN). Due to this, SNNs are better suited for real-life deployments. However, similar to ANNs, SNNs also benefit from deeper architectures to obtain improved performance. Furthermore, like the deep ANNs, the memory, compute and power requirements of SNNs also increase with model size, and model compression becomes a necessity. Knowledge distillation is a model compression technique that enables transferring the learning of a large machine learning model to a smaller model with minimal loss in performance. In this paper, we propose techniques for knowledge distillation in spiking neural networks for the task of image classification. We present ways to distill spikes from a larger SNN, also called the teacher network, to a smaller one, also called the student network, while minimally impacting the classification accuracy. We demonstrate the effectiveness of the proposed method with detailed experiments on three standard datasets while proposing novel distillation methodologies and loss functions. We also present a multi-stage knowledge distillation technique for SNNs using an intermediate network to obtain higher performance from the student network. Our approach is expected to open up new avenues for deploying high performing large SNN models on resource-constrained hardware platforms.