# Advancing Spiking Neural Networks towards Deep Residual Learning

arXiv: https://arxiv.org/abs/2112.08954 · HTML: https://mlsanity.com/paper/2112.08954/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 75 references that are in the map.
- First posted on arXiv 15 Dec 2021.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Deep Residual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2102.04159/) (2021)
- [Going Deeper With Directly-Trained Larger Spiking Neural Networks](https://mlsanity.com/paper/2011.05280/) (2020)
- [Going Deeper in Spiking Neural Networks: VGG and Residual Architectures](https://mlsanity.com/paper/1802.02627/) (2018)
- [Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm](https://mlsanity.com/paper/1808.00278/) (2018)
- [Direct Training for Spiking Neural Networks: Faster, Larger, Better](https://mlsanity.com/paper/1809.05793/) (2018)
- [Training Deep Spiking Neural Networks using Backpropagation](https://mlsanity.com/paper/1608.08782/) (2016)
- [Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection](https://mlsanity.com/paper/1903.06530/) (2019)
- [Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks](https://mlsanity.com/paper/1806.05393/) (2018)
- [RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Network](https://mlsanity.com/paper/2003.01811/) (2020)
- [Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks](https://mlsanity.com/paper/2103.00476/) (2021)
- [A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibration](https://mlsanity.com/paper/2106.06984/) (2021)
- [Training Binary Neural Networks with Real-to-Binary Convolutions](https://mlsanity.com/paper/2003.11535/) (2020)
- [Temporal-wise Attention Spiking Neural Networks for Event Streams Classification](https://mlsanity.com/paper/2107.11711/) (2021)
- [Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks](https://mlsanity.com/paper/2106.01862/) (2021)
- [Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks](https://mlsanity.com/paper/2003.06696/) (2020)
- [Temporal-Coded Deep Spiking Neural Network with Easy Training and Robust Performance](https://mlsanity.com/paper/1909.10837/) (2019)
- [DART: Distribution Aware Retinal Transform for Event-based Cameras](https://mlsanity.com/paper/1710.10800/) (2017)

## Built on by (most cited first, 30 of 79)

- [Spike-driven Transformer](https://mlsanity.com/paper/2307.01694/) (2023)
- [Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips](https://mlsanity.com/paper/2404.03663/) (2024)
- [Deep Directly-Trained Spiking Neural Networks for Object Detection](https://mlsanity.com/paper/2307.11411/) (2023)
- [Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training](https://mlsanity.com/paper/2411.16061/) (2024)
- [Spiking Vision Transformer with Saccadic Attention](https://mlsanity.com/paper/2502.12677/) (2025)
- [Spike2Former: Efficient Spiking Transformer for High-performance Image Segmentation](https://mlsanity.com/paper/2412.14587/) (2024)
- [Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks](https://mlsanity.com/paper/2401.04486/) (2024)
- [Inherent Redundancy in Spiking Neural Networks](https://mlsanity.com/paper/2308.08227/) (2023)
- [Autonomous Driving with Spiking Neural Networks](https://mlsanity.com/paper/2405.19687/) (2024)
- [Quantized Spike-driven Transformer](https://mlsanity.com/paper/2501.13492/) (2025)
- [TIM: An Efficient Temporal Interaction Module for Spiking Transformer](https://mlsanity.com/paper/2401.11687/) (2024)
- [LM-HT SNN: Enhancing the Performance of SNN to ANN Counterpart through Learnable Multi-hierarchical Threshold Model](https://mlsanity.com/paper/2402.00411/) (2024)
- [Efficient 3D Recognition with Event-driven Spike Sparse Convolution](https://mlsanity.com/paper/2412.07360/) (2024)
- [DA-LIF: Dual Adaptive Leaky Integrate-and-Fire Model for Deep Spiking Neural Networks](https://mlsanity.com/paper/2502.10422/) (2025)
- [Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review](https://mlsanity.com/paper/2502.14886/) (2025)
- [Advancing Spiking Neural Networks towards Multiscale Spatiotemporal Interaction Learning](https://mlsanity.com/paper/2405.13672/) (2024)
- [Toward Large-scale Spiking Neural Networks: A Comprehensive Survey and Future Directions](https://mlsanity.com/paper/2409.02111/) (2024)
- [Memory-Efficient Reversible Spiking Neural Networks](https://mlsanity.com/paper/2312.07922/) (2023)
- [Spiking Meets Attention: Efficient Remote Sensing Image Super-Resolution with Attention Spiking Neural Networks](https://mlsanity.com/paper/2503.04223/) (2025)
- [MK-SGN: A Spiking Graph Convolutional Network with Multimodal Fusion and Knowledge Distillation for Skeleton-based Action Recognition](https://mlsanity.com/paper/2404.10210/) (2024)
- [TS-SNN: Temporal Shift Module for Spiking Neural Networks](https://mlsanity.com/paper/2505.04165/) (2025)
- [SDiT: Spiking Diffusion Model with Transformer](https://mlsanity.com/paper/2402.11588/) (2024)
- [SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and O(T) Complexity](https://mlsanity.com/paper/2505.10352/) (2025)
- [Recent Advances in Scalable Energy-Efficient and Trustworthy Spiking Neural networks: from Algorithms to Technology](https://mlsanity.com/paper/2312.01213/) (2023)
- [ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition](https://mlsanity.com/paper/2409.01564/) (2024)
- [SpikeBottleNet: Spike-Driven Feature Compression Architecture for Edge-Cloud Co-Inference](https://mlsanity.com/paper/2410.08673/) (2024)
- [Exploiting Label Skewness for Spiking Neural Networks in Federated Learning](https://mlsanity.com/paper/2412.17305/) (2024)
- [Spiking Neural Networks Need High Frequency Information](https://mlsanity.com/paper/2505.18608/) (2025)
- [I2E: Real-Time Image-to-Event Conversion for High-Performance Spiking Neural Networks](https://mlsanity.com/paper/2511.08065/) (2025)
- [SynA-ResNet: Spike-driven ResNet Achieved through OR Residual Connection](https://mlsanity.com/paper/2311.06570/) (2023)

## Abstract (arXiv)

Despite the rapid progress of neuromorphic computing, inadequate capacity and insufficient representation power of spiking neural networks (SNNs) severely restrict their application scope in practice. Residual learning and shortcuts have been evidenced as an important approach for training deep neural networks, but rarely did previous work assess their applicability to the characteristics of spike-based communication and spatiotemporal dynamics. In this paper, we first identify that this negligence leads to impeded information flow and the accompanying degradation problem in previous residual SNNs. To address this issue, we propose a novel SNN-oriented residual architecture termed MS-ResNet, which establishes membrane-based shortcut pathways, and further prove that the gradient norm equality can be achieved in MS-ResNet by introducing block dynamical isometry theory, which ensures the network can be well-behaved in a depth-insensitive way. Thus we are able to significantly extend the depth of directly trained SNNs, e.g., up to 482 layers on CIFAR-10 and 104 layers on ImageNet, without observing any slight degradation problem. To validate the effectiveness of MS-ResNet, experiments on both frame-based and neuromorphic datasets are conducted. MS-ResNet104 achieves a superior result of 76.02% accuracy on ImageNet, which is the highest to our best knowledge in the domain of directly trained SNNs. Great energy efficiency is also observed, with an average of only one spike per neuron needed to classify an input sample. We believe our powerful and scalable models will provide a strong support for further exploration of SNNs.