mlsanityData to 7 Oct 2026

Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training

arXiv 2510.06254 ↗First posted 4 Oct 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Spiking Neural Networks (SNNs) exhibit exceptional energy efficiency on neuromorphic hardware due to their sparse activation patterns. However, conventional training methods based on surrogate gradients and Backpropagation Through Time (BPTT) not only lag behind Artificial Neural Networks (ANNs) in performance, but also incur significant computational and memory overheads that grow linearly with the temporal dimension. To enable high-performance SNN training under limited computational resources, we propose an enhanced self-distillation framework, jointly optimized with rate-based backpropagation. Specifically, the firing rates of intermediate SNN layers are projected onto lightweight ANN branches, and high-quality knowledge generated by the model itself is used to optimize substructures through the ANN pathways. Unlike traditional self-distillation paradigms, we observe that low-quality self-generated knowledge may hinder convergence. To address this, we decouple the teacher signal into reliable and unreliable components, ensuring that only reliable knowledge is used to guide the optimization of the model. Extensive experiments on CIFAR-10, CIFAR-100, CIFAR10-DVS, and ImageNet demonstrate that our method reduces training complexity while achieving high-performance SNN training. Our code is available at https://github.com/Intelli-Chip-Lab/enhanced-self-distillation-framework-for-snn.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 49 of its 86 references that are in the map. First posted on arXiv 4 Oct 2025.

Builds on

49 of 86 references.

References · 49

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 604
  5. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  6. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  7. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 399
  8. 2022Decoupled Knowledge Distillationcited by 380
  9. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 340
  10. 2021Deep Residual Learning in Spiking Neural Networkscited by 326
  11. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
  12. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
37 more
  1. 2020Going Deeper With Directly-Trained Larger Spiking Neural Networkscited by 268
  2. 2018SLAYER: Spike Layer Error Reassignment in Timecited by 257
  3. 2018Going Deeper in Spiking Neural Networks: VGG and Residual Architecturescited by 236
  4. 2022Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingcited by 192
  5. 2017SuperSpike: Supervised learning in multi-layer spiking neural networkscited by 178
  6. 2023SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligencecited by 163
  7. 2020RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Networkcited by 141
  8. 2021Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networkscited by 134
  9. 2023Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networkscited by 116
  10. 2021A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibrationcited by 111
  11. 2020Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networkscited by 98
  12. 2022Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representationcited by 89
  13. 2016Supervised learning based on temporal coding in spiking neural networkscited by 75
  14. 2021Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networkscited by 74
  15. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
  16. 2022Online Training Through Time for Spiking Neural Networkscited by 48
  17. 2021Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillationcited by 46
  18. 2020Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networkscited by 43
  19. 2023Constructing Deep Spiking Neural Networks from Artificial Neural Networks with Knowledge Distillationcited by 41
  20. 2021Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium Statecited by 39
  21. 2023Reducing ANN-SNN Conversion Error through Residual Membrane Potentialcited by 39
  22. 2023SEENN: Towards Temporal Spiking Early-Exit Neural Networkscited by 39
  23. 2023Bridging the Gap between ANNs and SNNs by Calibrating Offset Spikescited by 38
  24. 2023Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networkscited by 37
  25. 2023Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorizationcited by 34
  26. 2021Accurate online training of dynamical spiking neural networks through Forward Propagation Through Timecited by 33
  27. 2020Online Spatio-Temporal Learning in Deep Neural Networkscited by 32
  28. 2020Unifying Activation- and Timing-based Learning Rules for Spiking Neural Networkscited by 27
  29. 2020Distilling Spikes: Knowledge Distillation in Spiking Neural Networkscited by 24
  30. 2021Backpropagated Neighborhood Aggregation for Accurate Training of Spiking Neural Networkscited by 14
  31. 2023LaSNN: Layer-wise ANN-to-SNN Distillation for Effective and Efficient Training in Deep Spiking Neural Networkscited by 11
  32. 2023Uncovering the Representation of Spiking Neural Networks Trained with Surrogate Gradientcited by 11
  33. 2024BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge Distillationcited by 11
  34. 2021Energy-efficient Knowledge Distillation for Spiking Neural Networkscited by 10
  35. 2023Temporal Knowledge Sharing enable Spiking Neural Network Learning from Past and Futurecited by 10
  36. 2024Advancing Training Efficiency of Deep Spiking Neural Networks through Rate-based Backpropagationcited by 8
  37. 2025Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacementcited by 6

Built on by

Not cited yet.