mlsanityData to 14 Oct 2026

Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacement

arXiv 2503.16572 ↗First posted 20 Mar 2025cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Spiking Neural Networks (SNNs) have garnered considerable attention as a potential alternative to Artificial Neural Networks (ANNs). Recent studies have highlighted SNNs' potential on large-scale datasets. For SNN training, two main approaches exist: direct training and ANN-to-SNN (ANN2SNN) conversion. To fully leverage existing ANN models in guiding SNN learning, either direct ANN-to-SNN conversion or ANN-SNN distillation training can be employed. In this paper, we propose an ANN-SNN distillation framework from the ANN-to-SNN perspective, designed with a block-wise replacement strategy for ANN-guided learning. By generating intermediate hybrid models that progressively align SNN feature spaces to those of ANN through rate-based features, our framework naturally incorporates rate-based backpropagation as a training method. Our approach achieves results comparable to or better than state-of-the-art SNN distillation methods, showing both training and learning efficiency.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 40 of its 71 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

40 of 71 references.

References · 40

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  5. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  6. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 626
  7. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  8. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 406
  9. 2021Deep Residual Learning in Spiking Neural Networkscited by 331
  10. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  11. 2019Structured Knowledge Distillation for Dense Predictioncited by 279
  12. 2020Going Deeper With Directly-Trained Larger Spiking Neural Networkscited by 278
28 more
  1. 2018SLAYER: Spike Layer Error Reassignment in Timecited by 266
  2. 2022Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingcited by 198
  3. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
  4. 2023Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networkscited by 122
  5. 2020Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networkscited by 100
  6. 2022Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representationcited by 89
  7. 2021HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noisecited by 65
  8. 2015Spiking Deep Networks with LIF Neuronscited by 63
  9. 2022GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networkscited by 63
  10. 2020LIAF-Net: Leaky Integrate and Analog Fire Network for Lightweight and Efficient Spatiotemporal Information Processingcited by 56
  11. 2022Online Training Through Time for Spiking Neural Networkscited by 49
  12. 2020Inherent Adversarial Robustness of Deep Spiking Neural Networks: Effects of Discrete Input Encoding and Non-Linear Activationscited by 48
  13. 2023Fast-SNN: Fast Spiking Neural Network by Converting Quantized ANNcited by 46
  14. 2023Constructing Deep Spiking Neural Networks from Artificial Neural Networks with Knowledge Distillationcited by 44
  15. 2022Optimized Potential Initialization for Low-latency Spiking Neural Networkscited by 41
  16. 2021Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium Statecited by 40
  17. 2023Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorizationcited by 35
  18. 2020Unifying Activation- and Timing-based Learning Rules for Spiking Neural Networkscited by 28
  19. 2020Distilling Spikes: Knowledge Distillation in Spiking Neural Networkscited by 27
  20. 2020Securing Deep Spiking Neural Networks against Adversarial Attacks through Inherent Structural Parameterscited by 27
  21. 2022Real Spike: Learning Real-valued Spikes for Spiking Neural Networkscited by 22
  22. 2023LaSNN: Layer-wise ANN-to-SNN Distillation for Effective and Efficient Training in Deep Spiking Neural Networkscited by 12
  23. 2021Energy-efficient Knowledge Distillation for Spiking Neural Networkscited by 11
  24. 2023Uncovering the Representation of Spiking Neural Networks Trained with Surrogate Gradientcited by 11
  25. 2024Self-Distillation Learning Based on Temporal-Spatial Consistency for Spiking Neural Networkscited by 9
  26. 2024Advancing Training Efficiency of Deep Spiking Neural Networks through Rate-based Backpropagationcited by 8
  27. 2024Decoupling Dark Knowledge via Block-wise Logit Distillation for Feature-level Alignmentcited by 7
  28. 2024Improving Quantization-aware Training of Low-Precision Network via Block Replacement on Full-Precision Counterpartcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Edge Intelligence with Spiking Neural Networkscited by 2
  2. 2025Spiking Vocos: An Energy-Efficient Neural Vocodercited by 0
  3. 2025Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Trainingcited by 0
  4. 2025Stabilizing Direct Training of Spiking Neural Networks: Membrane Potential Initialization and Threshold-robust Surrogate Gradientcited by 0
  5. 2026Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networkscited by 0
  6. 2026STARS: Spike Tail-Aware Relational Synthesis for ANN-to-SNN Data-Free Knowledge Distillationcited by 0