# Training Spiking Neural Networks Using Lessons From Deep Learning

arXiv: https://arxiv.org/abs/2109.12894 · HTML: https://mlsanity.com/paper/2109.12894/

## Key facts

- Cited by 259 papers in the map (data through 14 Oct 2026).
- Builds on 75 of its 273 references that are in the map.
- First posted on arXiv 27 Sep 2021.

## Builds on (most cited first, 30 of 75)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [Deep Residual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2102.04159/) (2021)
- [Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades](https://mlsanity.com/paper/1507.07629/) (2015)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception](https://mlsanity.com/paper/1801.10202/) (2018)
- [Training Deep Spiking Neural Networks using Backpropagation](https://mlsanity.com/paper/1608.08782/) (2016)
- [The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM](https://mlsanity.com/paper/1610.08336/) (2016)
- [The Computational Limits of Deep Learning](https://mlsanity.com/paper/2007.05558/) (2020)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)

## Built on by (most cited first, 30 of 259)

- [SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence](https://mlsanity.com/paper/2310.16620/) (2023)
- [SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks](https://mlsanity.com/paper/2302.13939/) (2023)
- [Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training](https://mlsanity.com/paper/2411.16061/) (2024)
- [Fast-SNN: Fast Spiking Neural Network by Converting Quantized ANN](https://mlsanity.com/paper/2305.19868/) (2023)
- [TCJA-SNN: Temporal-Channel Joint Attention for Spiking Neural Networks](https://mlsanity.com/paper/2206.10177/) (2022)
- [NeuroBench: A Framework for Benchmarking Neuromorphic Computing Algorithms and Systems](https://mlsanity.com/paper/2304.04640/) (2023)
- [Brain-inspired learning in artificial neural networks: a review](https://mlsanity.com/paper/2305.11252/) (2023)
- [Green Edge AI: A Contemporary Survey](https://mlsanity.com/paper/2312.00333/) (2023)
- [SpiNNaker2: A Large-Scale Neuromorphic System for Event-Based and Asynchronous Machine Learning](https://mlsanity.com/paper/2401.04491/) (2024)
- [Scalable MatMul-free Language Modeling](https://mlsanity.com/paper/2406.02528/) (2024)
- [Signal Propagation: A Framework for Learning and Inference In a Forward Pass](https://mlsanity.com/paper/2204.01723/) (2022)
- [Learning Long Sequences in Spiking Neural Networks](https://mlsanity.com/paper/2401.00955/) (2023)
- [Spiking Neural Networks for Frame-based and Event-based Single Object Localization](https://mlsanity.com/paper/2206.06506/) (2022)
- [Sneaky Spikes: Uncovering Stealthy Backdoor Attacks in Spiking Neural Networks with Neuromorphic Data](https://mlsanity.com/paper/2302.06279/) (2023)
- [SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillation](https://mlsanity.com/paper/2308.15122/) (2023)
- [Inherent Redundancy in Spiking Neural Networks](https://mlsanity.com/paper/2308.08227/) (2023)
- [Autonomous Driving with Spiking Neural Networks](https://mlsanity.com/paper/2405.19687/) (2024)
- [Reconsidering the Energy Efficiency of Spiking Neural Networks Inference from Analytical Perspectives](https://mlsanity.com/paper/2409.08290/) (2024)
- [Quantized Spike-driven Transformer](https://mlsanity.com/paper/2501.13492/) (2025)
- [Sub-mW Neuromorphic SNN audio processing applications with Rockpool and Xylo](https://mlsanity.com/paper/2208.12991/) (2022)
- [Spiking neural networks for nonlinear regression](https://mlsanity.com/paper/2210.03515/) (2022)
- [Neuromorphic Computing and Sensing in Space](https://mlsanity.com/paper/2212.05236/) (2022)
- [Spiking-Fer: Spiking Neural Network for Facial Expression Recognition With Event Cameras](https://mlsanity.com/paper/2304.10211/) (2023)
- [To Spike or Not To Spike: A Digital Hardware Perspective on Deep Learning Acceleration](https://mlsanity.com/paper/2306.15749/) (2023)
- [A Survey on Multimodal Wearable Sensor-based Human Action Recognition](https://mlsanity.com/paper/2404.15349/) (2024)
- [Spiking Convolutional Neural Networks for Text Classification](https://mlsanity.com/paper/2406.19230/) (2024)
- [FSTA-SNN:Frequency-based Spatial-Temporal Attention Module for Spiking Neural Networks](https://mlsanity.com/paper/2501.14744/) (2024)
- [Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networks](https://mlsanity.com/paper/2503.03144/) (2025)
- [Applications of Spiking Neural Networks in Visual Place Recognition](https://mlsanity.com/paper/2311.13186/) (2023)
- [SPikE-SSM: A Sparse, Precise, and Efficient Spiking State Space Model for Long Sequences Learning](https://mlsanity.com/paper/2410.17268/) (2024)

## Abstract (arXiv)

The brain is the perfect place to look for inspiration to develop more efficient neural networks. The inner workings of our synapses and neurons provide a glimpse at what the future of deep learning might look like. This paper serves as a tutorial and perspective showing how to apply the lessons learnt from several decades of research in deep learning, gradient descent, backpropagation and neuroscience to biologically plausible spiking neural neural networks. We also explore the delicate interplay between encoding data as spikes and the learning process; the challenges and solutions of applying gradient-based learning to spiking neural networks (SNNs); the subtle link between temporal backpropagation and spike timing dependent plasticity, and how deep learning might move towards biologically plausible online learning. Some ideas are well accepted and commonly used amongst the neuromorphic engineering community, while others are presented or justified for the first time here. The fields of deep learning and spiking neural networks evolve very rapidly. We endeavour to treat this document as a 'dynamic' manuscript that will continue to be updated as the common practices in training SNNs also change. A series of companion interactive tutorials complementary to this paper using our Python package, snnTorch, are also made available. See https://snntorch.readthedocs.io/en/latest/tutorials/index.html .