mlsanityData to 7 Oct 2026

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment

arXiv 2501.15925 ↗First posted 27 Jan 2025cs.LG q-bio.NCcited by 2

Not on the map. Why?

Abstract (arXiv)

Spiking Neural Networks (SNNs) are emerging as a brain-inspired alternative to traditional Artificial Neural Networks (ANNs), prized for their potential energy efficiency on neuromorphic hardware. Despite this, SNNs often suffer from accuracy degradation compared to ANNs and face deployment challenges due to fixed inference timesteps, which require retraining for adjustments, limiting operational flexibility. To address these issues, our work considers the spatio-temporal property inherent in SNNs, and proposes a novel distillation framework for deep SNNs that optimizes performance across full-range timesteps without specific retraining, enhancing both efficacy and deployment adaptability. We provide both theoretical analysis and empirical validations to illustrate that training guarantees the convergence of all implicit models across full-range timesteps. Experimental results on CIFAR-10, CIFAR-100, CIFAR10-DVS, and ImageNet demonstrate state-of-the-art performance among distillation-based SNNs training methods. Our code is available at https://github.com/Intelli-Chip-Lab/snntemporaldecouplingdistillation.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 64 of its 105 references that are in the map. First posted on arXiv 27 Jan 2025.

Builds on

64 of 105 references.

References · 64

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  6. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  7. 2020Knowledge Distillation: A Surveycited by 1,578
  8. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 604
  9. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  10. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  11. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 399
  12. 2021ResNet strikes back: An improved training procedure in timmcited by 365
38 more
  1. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 340
  2. 2021Deep Residual Learning in Spiking Neural Networkscited by 326
  3. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
  4. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  5. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  6. 2020Going Deeper With Directly-Trained Larger Spiking Neural Networkscited by 268
  7. 2018SLAYER: Spike Layer Error Reassignment in Timecited by 257
  8. 2018Going Deeper in Spiking Neural Networks: VGG and Residual Architecturescited by 236
  9. 2016Training Deep Spiking Neural Networks using Backpropagationcited by 203
  10. 2022Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingcited by 192
  11. 2017SuperSpike: Supervised learning in multi-layer spiking neural networkscited by 178
  12. 2023SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligencecited by 163
  13. 2020RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Networkcited by 141
  14. 2021Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networkscited by 134
  15. 2023Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networkscited by 116
  16. 2021A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibrationcited by 111
  17. 2021Temporal-wise Attention Spiking Neural Networks for Event Streams Classificationcited by 106
  18. 2020Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networkscited by 98
  19. 2022Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representationcited by 89
  20. 2021Advancing Spiking Neural Networks towards Deep Residual Learningcited by 76
  21. 2016Supervised learning based on temporal coding in spiking neural networkscited by 75
  22. 2021Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networkscited by 74
  23. 2022GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networkscited by 63
  24. 2024Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Trainingcited by 57
  25. 2019Temporal-Coded Deep Spiking Neural Network with Easy Training and Robust Performancecited by 45
  26. 2020Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networkscited by 43
  27. 2023Constructing Deep Spiking Neural Networks from Artificial Neural Networks with Knowledge Distillationcited by 41
  28. 2021Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium Statecited by 39
  29. 2023Reducing ANN-SNN Conversion Error through Residual Membrane Potentialcited by 39
  30. 2023SEENN: Towards Temporal Spiking Early-Exit Neural Networkscited by 39
  31. 2023Bridging the Gap between ANNs and SNNs by Calibrating Offset Spikescited by 38
  32. 2023Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networkscited by 37
  33. 2023Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorizationcited by 34
  34. 2021Accurate online training of dynamical spiking neural networks through Forward Propagation Through Timecited by 33
  35. 2020Online Spatio-Temporal Learning in Deep Neural Networkscited by 32
  36. 2023Direct Learning-Based Deep Spiking Neural Networks: A Reviewcited by 31
  37. 2020Unifying Activation- and Timing-based Learning Rules for Spiking Neural Networkscited by 27
  38. 2023Gated Attention Coding for Training High-performance and Efficient Spiking Neural Networkscited by 26

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025CogniSNN: Enabling Neuron-Expandability, Pathway-Reusability, and Dynamic-Configurability with Random Graph Architectures in Spiking Neural Networkscited by 0
  2. 2026Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networkscited by 0