mlsanityData to 7 Oct 2026

Constructing Deep Spiking Neural Networks from Artificial Neural Networks with Knowledge Distillation

arXiv 2304.05627 ↗First posted 12 Apr 2023cs.NE cs.AIcited by 41

Not on the map. Why?

Abstract (arXiv)

Spiking neural networks (SNNs) are well known as the brain-inspired models with high computing efficiency, due to a key component that they utilize spikes as information units, close to the biological neural systems. Although spiking based models are energy efficient by taking advantage of discrete spike signals, their performance is limited by current network structures and their training methods. As discrete signals, typical SNNs cannot apply the gradient descent rules directly into parameters adjustment as artificial neural networks (ANNs). Aiming at this limitation, here we propose a novel method of constructing deep SNN models with knowledge distillation (KD) that uses ANN as teacher model and SNN as student model. Through ANN-SNN joint training algorithm, the student SNN model can learn rich feature information from the teacher ANN model through the KD method, yet it avoids training SNN from scratch when communicating with non-differentiable spikes. Our method can not only build a more efficient deep spiking structure feasibly and reasonably, but use few time steps to train whole model compared to direct training or ANN to SNN methods. More importantly, it has a superb ability of noise immunity for various types of artificial noises and natural signals. The proposed novel method provides efficient ways to improve the performance of SNN through constructing deeper structures in a high-throughput fashion, with potential usage for light and efficient brain-inspired computing of practical scenarios.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 18 of its 37 references that are in the map. First posted on arXiv 12 Apr 2023.

Builds on

18 of 37 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  4. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 604
  5. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 399
  6. 2019On the Efficacy of Knowledge Distillationcited by 377
  7. 2021Deep Residual Learning in Spiking Neural Networkscited by 326
  8. 2018Deep Learning in Spiking Neural Networkscited by 258
  9. 2018Going Deeper in Spiking Neural Networks: VGG and Residual Architecturescited by 236
  10. 2016Deep Pyramidal Residual Networkscited by 163
  11. 2020Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagationcited by 142
  12. 2020RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Networkcited by 141
6 more
  1. 2021Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networkscited by 134
  2. 2023Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networkscited by 116
  3. 2021Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networkscited by 74
  4. 2020Unifying Activation- and Timing-based Learning Rules for Spiking Neural Networkscited by 27
  5. 2016STDP-based spiking deep convolutional neural networks for object recognitioncited by 24
  6. 2020Distilling Spikes: Knowledge Distillation in Spiking Neural Networkscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 10; 2025: 19; 2026: 5.

41 papers.

Sort:

  1. 2023Ternary Spike: Learning Ternary Spikes for Spiking Neural Networkscited by 36
  2. 2023Membrane Potential Batch Normalization for Spiking Neural Networkscited by 35
  3. 2023Spiking PointNet: Spiking Neural Networks for Point Cloudscited by 34
  4. 2023Direct Learning-Based Deep Spiking Neural Networks: A Reviewcited by 31
  5. 2023RMP-Loss: Regularizing Membrane Potential Distribution for Spiking Neural Networkscited by 27
  6. 2024Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networkscited by 14
  7. 2025Quantized Spike-driven Transformercited by 12
  8. 2023Biologically inspired structure learning with reverse knowledge distillation for spiking neural networkscited by 11
  9. 2024Fourier Prompt Tuning for Modality-Incomplete Scene Segmentationcited by 11
  10. 2024FSTA-SNN:Frequency-based Spatial-Temporal Attention Module for Spiking Neural Networkscited by 10
  11. 2025Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networkscited by 10
  12. 2024Self-Distillation Learning Based on Temporal-Spatial Consistency for Spiking Neural Networkscited by 9
29 more
  1. 2025DA-LIF: Dual Adaptive Leaky Integrate-and-Fire Model for Deep Spiking Neural Networkscited by 8
  2. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 7
  3. 2024Toward Large-scale Spiking Neural Networks: A Comprehensive Survey and Future Directionscited by 6
  4. 2025Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacementcited by 6
  5. 2025TS-SNN: Temporal Shift Module for Spiking Neural Networkscited by 5
  6. 2024Apprenticeship-Inspired Elegance: Synergistic Knowledge Distillation Empowers Spiking Neural Networks for Efficient Single-Eye Emotion Recognitioncited by 3
  7. 2025SOTA: Spike-Navigated Optimal TrAnsport Saliency Region Detection in Composite-bias Videoscited by 3
  8. 2025Spiking Neural Networks Need High Frequency Informationcited by 3
  9. 2025ReverB-SNN: Reversing Bit of the Weight and Activation for Spiking Neural Networkscited by 3
  10. 2025Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deploymentcited by 2
  11. 2025Edge Intelligence with Spiking Neural Networkscited by 2
  12. 2025Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillerscited by 2
  13. 2024Computer Vision Model Compression Techniques for Embedded Systems: A Surveycited by 1
  14. 2025Continual Learning of Multiple Cognitive Functions with Brain-inspired Temporal Development Mechanismcited by 1
  15. 2025Head-Tail-Aware KL Divergence in Knowledge Distillation for Spiking Neural Networkscited by 1
  16. 2026BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillationcited by 1
  17. 2023Rotation Augmented Distillation for Exemplar-Free Class Incremental Learning with Detailed Analysiscited by 0
  18. 2024On Reducing Activity with Distillation and Regularization for Energy Efficient Spiking Neural Networkscited by 0
  19. 2024Toward Efficient Deep Spiking Neuron Networks:A Survey On Compressioncited by 0
  20. 2025Spiking Brain Compression: Post-Training Second-order Compression for Spiking Neural Networkscited by 0
  21. 2025SDSNN: A Single-Timestep Spiking Neural Network with Self-Dropping Neuron and Bayesian Optimizationcited by 0
  22. 2025Incorporating the Refractory Period into Spiking Neural Networks through Spike-Triggered Threshold Dynamicscited by 0
  23. 2025Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Trainingcited by 0
  24. 2025A Scalable, Causal, and Energy Efficient Framework for Neural Decoding with Spiking Neural Networkscited by 0
  25. 2025A Closer Look at Knowledge Distillation in Spiking Neural Network Trainingcited by 0
  26. 2026Stable Spike: Dual Consistency Optimization via Bitwise AND Operations for Spiking Neural Networkscited by 0
  27. 2026Ge2mS-T: Multi-Dimensional Grouping for Ultra-High Energy Efficiency in Spiking Transformercited by 0
  28. 2026Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networkscited by 0
  29. 2026Active Spiking Perception: The Membrane Potential as a Belief State for Anytime 3D Point Cloud Recognitioncited by 0