# LaSNN: Layer-wise ANN-to-SNN Distillation for Effective and Efficient Training in Deep Spiking Neural Networks

arXiv: https://arxiv.org/abs/2304.09101 · HTML: https://mlsanity.com/paper/2304.09101/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 17 Apr 2023.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Deep Learning in Spiking Neural Networks](https://mlsanity.com/paper/1804.08150/) (2018)
- [Going Deeper in Spiking Neural Networks: VGG and Residual Architectures](https://mlsanity.com/paper/1802.02627/) (2018)
- [Direct Training for Spiking Neural Networks: Faster, Larger, Better](https://mlsanity.com/paper/1809.05793/) (2018)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)
- [RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Network](https://mlsanity.com/paper/2003.01811/) (2020)
- [Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation](https://mlsanity.com/paper/2005.01807/) (2020)
- [Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks](https://mlsanity.com/paper/2103.00476/) (2021)
- [A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibration](https://mlsanity.com/paper/2106.06984/) (2021)
- [Distilling Spikes: Knowledge Distillation in Spiking Neural Networks](https://mlsanity.com/paper/2005.00288/) (2020)
- [Visual Explanations from Spiking Neural Networks using Interspike Intervals](https://mlsanity.com/paper/2103.14441/) (2021)
- [Energy-efficient Knowledge Distillation for Spiking Neural Networks](https://mlsanity.com/paper/2106.07172/) (2021)
- [Bio-Inspired Spiking Convolutional Neural Network using Layer-wise Sparse Coding and STDP Learning](https://mlsanity.com/paper/1611.03000/) (2016)
- [TrueHappiness: Neuromorphic Emotion Recognition on TrueNorth](https://mlsanity.com/paper/1601.04183/) (2016)

## Built on by (most cited first, 12 of 12)

- [BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge Distillation](https://mlsanity.com/paper/2407.09083/) (2024)
- [Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networks](https://mlsanity.com/paper/2503.03144/) (2025)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacement](https://mlsanity.com/paper/2503.16572/) (2025)
- [Cross Knowledge Distillation between Artificial and Spiking Neural Networks](https://mlsanity.com/paper/2507.09269/) (2025)
- [Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment](https://mlsanity.com/paper/2501.15925/) (2025)
- [Energy-Efficient and Dequantization-Free Q-LLMs: A Spiking Neural Network Approach to Salient Value Mitigation](https://mlsanity.com/paper/2510.19498/) (2025)
- [SpikeProphecy: A Large-Scale Benchmark for Autoregressive Neural Population Forecasting](https://mlsanity.com/paper/2605.12992/) (2026)
- [Head-Tail-Aware KL Divergence in Knowledge Distillation for Spiking Neural Networks](https://mlsanity.com/paper/2504.20445/) (2025)
- [Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training](https://mlsanity.com/paper/2510.06254/) (2025)
- [Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networks](https://mlsanity.com/paper/2605.14252/) (2026)
- [STARS: Spike Tail-Aware Relational Synthesis for ANN-to-SNN Data-Free Knowledge Distillation](https://mlsanity.com/paper/2605.27409/) (2026)

## Abstract (arXiv)

Spiking Neural Networks (SNNs) are biologically realistic and practically promising in low-power computation because of their event-driven mechanism. Usually, the training of SNNs suffers accuracy loss on various tasks, yielding an inferior performance compared with ANNs. A conversion scheme is proposed to obtain competitive accuracy by mapping trained ANNs' parameters to SNNs with the same structures. However, an enormous number of time steps are required for these converted SNNs, thus losing the energy-efficient benefit. Utilizing both the accuracy advantages of ANNs and the computing efficiency of SNNs, a novel SNN training framework is proposed, namely layer-wise ANN-to-SNN knowledge distillation (LaSNN). In order to achieve competitive accuracy and reduced inference latency, LaSNN transfers the learning from a well-trained ANN to a small SNN by distilling the knowledge other than converting the parameters of ANN. The information gap between heterogeneous ANN and SNN is bridged by introducing the attention scheme, the knowledge in an ANN is effectively compressed and then efficiently transferred by utilizing our layer-wise distillation paradigm. We conduct detailed experiments to demonstrate the effectiveness, efficacy, and scalability of LaSNN on three benchmark data sets (CIFAR-10, CIFAR-100, and Tiny ImageNet). We achieve competitive top-1 accuracy compared to ANNs and 20x faster inference than converted SNNs with similar performance. More importantly, LaSNN is dexterous and extensible that can be effortlessly developed for SNNs with different architectures/depths and input encoding methods, contributing to their potential development.