mlsanityData to 14 Oct 2026

Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers

arXiv 2510.07924 ↗First posted 9 Oct 2025cs.LGcited by 2

On the map: Knowledge distillation → Distillation for spiking neural networks

Abstract (arXiv)

Brain-inspired spiking neural networks (SNNs) promise to be a low-power alternative to computationally intensive artificial neural networks (ANNs), although performance gaps persist. Recent studies have improved the performance of SNNs through knowledge distillation, but rely on large teacher models or introduce additional training overhead. In this paper, we show that SNNs can be naturally deconstructed into multiple submodels for efficient self-distillation. We treat each timestep instance of the SNN as a submodel and evaluate its output confidence, thus efficiently identifying the strong and the weak. Based on this strong and weak relationship, we propose two efficient self-distillation schemes: (1) Strong2Weak: During training, the stronger "teacher" guides the weaker "student", effectively improving overall performance. (2) Weak2Strong: The weak serve as the "teacher", distilling the strong in reverse with underlying dark knowledge, again yielding significant performance gains. For both distillation schemes, we offer flexible implementations such as ensemble, simultaneous, and cascade distillation. Experiments show that our method effectively improves the discriminability and overall performance of the SNN, while its adversarial robustness is also enhanced, benefiting from the stability brought by self-distillation. This ingeniously exploits the temporal properties of SNNs and provides insight into how to efficiently train high-performance SNNs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Distillation for spiking neural networks, one of 12 directions in Knowledge distillation. Builds on 47 of its 75 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

47 of 75 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge Distillation12 Jul
  2. 2024Self-Distillation Learning Based on Temporal-Spatial Consistency for Spiking Neural Networks12 Jun
  3. 2023Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorization3 May
  4. 2023Constructing Deep Spiking Neural Networks from Artificial Neural Networks with Knowledge Distillation12 Apr

Other references · 43

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  6. 2016Adversarial examples in the physical worldcited by 1,769
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  8. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  9. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 406
  10. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 345
  11. 2021Deep Residual Learning in Spiking Neural Networkscited by 331
  12. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 271
31 more
  1. 2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 252
  2. 2023SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligencecited by 169
  3. 2020Resolution Adaptive Networks for Efficient Inferencecited by 146
  4. 2023Spike-driven Transformercited by 146
  5. 2020RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Networkcited by 145
  6. 2024Logit Standardization in Knowledge Distillationcited by 102
  7. 2022Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representationcited by 89
  8. 2021Advancing Spiking Neural Networks towards Deep Residual Learningcited by 79
  9. 2023Deep Directly-Trained Spiking Neural Networks for Object Detectioncited by 77
  10. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
  11. 2022GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networkscited by 63
  12. 2024QKFormer: Hierarchical Spiking Transformer using Q-K Attentioncited by 49
  13. 2023SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiationcited by 48
  14. 2020Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networkscited by 47
  15. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  16. 2023SEENN: Towards Temporal Spiking Early-Exit Neural Networkscited by 41
  17. 2024SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networkscited by 40
  18. 2023RMP-Loss: Regularizing Membrane Potential Distribution for Spiking Neural Networkscited by 28
  19. 2020Distilling Spikes: Knowledge Distillation in Spiking Neural Networkscited by 27
  20. 2021Spiking Neural Networks with Improved Inherent Recurrence Dynamics for Sequential Learningcited by 27
  21. 2023Gated Attention Coding for Training High-performance and Efficient Spiking Neural Networkscited by 26
  22. 2022Exploring Temporal Information Dynamics in Spiking Neural Networkscited by 24
  23. 2022Training Spiking Neural Networks with Local Tandem Learningcited by 23
  24. 2024CLIF: Complementary Leaky Integrate-and-Fire Neuron for Spiking Neural Networkscited by 21
  25. 2024Shrinking Your TimeStep: Towards Low-Latency Neuromorphic Object Recognition with Spiking Neural Networkcited by 19
  26. 2024Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networkscited by 14
  27. 2025Rethinking Spiking Neural Networks from an Ensemble Learning Perspectivecited by 12
  28. 2025STAA-SNN: Spatial-Temporal Attention Aggregator for Spiking Neural Networkscited by 11
  29. 2023Temporal Knowledge Sharing enable Spiking Neural Network Learning from Past and Futurecited by 10
  30. 2023Efficient Deep Spiking Multi-Layer Perceptrons with Multiplication-Free Inferencecited by 7
  31. 2024Flexible ViG: Learning the Self-Saliency for Flexible Object Recognitioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Stable Spike: Dual Consistency Optimization via Bitwise AND Operations for Spiking Neural Networkscited by 0
  2. 2026Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training
  2. 2025A Closer Look at Knowledge Distillation in Spiking Neural Network Training
  3. 2025Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networks
  4. 2025Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
  5. 2021Energy-efficient Knowledge Distillation for Spiking Neural Networks
  6. 2024On Reducing Activity with Distillation and Regularization for Energy Efficient Spiking Neural Networks
  7. 2025Head-Tail-Aware KL Divergence in Knowledge Distillation for Spiking Neural Networks
  8. 2023LaSNN: Layer-wise ANN-to-SNN Distillation for Effective and Efficient Training in Deep Spiking Neural Networks