mlsanityData to 14 Oct 2026

Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks

arXiv 2302.14311 ↗First posted 28 Feb 2023cs.NE cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

Spiking Neural Networks (SNNs) are promising energy-efficient models for neuromorphic computing. For training the non-differentiable SNN models, the backpropagation through time (BPTT) with surrogate gradients (SG) method has achieved high performance. However, this method suffers from considerable memory cost and training time during training. In this paper, we propose the Spatial Learning Through Time (SLTT) method that can achieve high performance while greatly improving training efficiency compared with BPTT. First, we show that the backpropagation of SNNs through the temporal domain contributes just a little to the final calculated gradients. Thus, we propose to ignore the unimportant routes in the computational graph during backpropagation. The proposed method reduces the number of scalar multiplications and achieves a small memory occupation that is independent of the total time steps. Furthermore, we propose a variant of SLTT, called SLTT-K, that allows backpropagation only at K time steps, then the required number of scalar multiplications is further reduced and is independent of the total time steps. Experiments on both static and neuromorphic datasets demonstrate superior training efficiency and performance of our SLTT. In particular, our method achieves state-of-the-art accuracy on ImageNet, while the memory cost and training time are reduced by more than 70% and 50%, respectively, compared with BPTT.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 41 of its 77 references that are in the map. First posted on arXiv 28 Feb 2023.

Builds on

41 of 77 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  6. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  7. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 626
  8. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 406
  9. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  10. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 345
  11. 2021Deep Residual Learning in Spiking Neural Networkscited by 331
  12. 2020Going Deeper With Directly-Trained Larger Spiking Neural Networkscited by 278
29 more
  1. 2018SLAYER: Spike Layer Error Reassignment in Timecited by 266
  2. 2018Going Deeper in Spiking Neural Networks: VGG and Residual Architecturescited by 245
  3. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  4. 2018Direct Training for Spiking Neural Networks: Faster, Larger, Bettercited by 233
  5. 2022Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingcited by 198
  6. 2017SuperSpike: Supervised learning in multi-layer spiking neural networkscited by 182
  7. 2020RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Networkcited by 145
  8. 2020Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagationcited by 144
  9. 2021Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networkscited by 139
  10. 2021A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibrationcited by 117
  11. 2022Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representationcited by 89
  12. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 88
  13. 2021Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networkscited by 78
  14. 2016Supervised learning based on temporal coding in spiking neural networkscited by 76
  15. 2016Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computingcited by 70
  16. 2022Neuromorphic Data Augmentation for Training Spiking Neural Networkscited by 52
  17. 2022Online Training Through Time for Spiking Neural Networkscited by 49
  18. 2017Gradient Descent for Spiking Neural Networkscited by 48
  19. 2019Temporal-Coded Deep Spiking Neural Network with Easy Training and Robust Performancecited by 46
  20. 2021Sparse Spiking Gradient Descentcited by 41
  21. 2021Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium Statecited by 40
  22. 2020Surrogate gradients for analog neuromorphic computingcited by 38
  23. 2020Brain-Inspired Learning on Neuromorphic Substratescited by 37
  24. 2021Accurate online training of dynamical spiking neural networks through Forward Propagation Through Timecited by 34
  25. 2020Online Spatio-Temporal Learning in Deep Neural Networkscited by 32
  26. 2022Training Spiking Neural Networks with Local Tandem Learningcited by 23
  27. 2021One Timestep is All You Need: Training Spiking Neural Networks with Ultra Low Latencycited by 17
  28. 2019SpikeGrad: An ANN-equivalent Computation Model for Implementing Backpropagation with Spikescited by 16
  29. 2023SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networkscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 21; 2026: 4.

37 papers.

Sort:

By area:

  1. 2024Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Trainingcited by 59
  2. 2025Rethinking Spiking Neural Networks from an Ensemble Learning Perspectivecited by 12
  3. 2024LM-HT SNN: Enhancing the Performance of SNN to ANN Counterpart through Learnable Multi-hierarchical Threshold Modelcited by 11
  4. 2025Differential Coding for Training-Free ANN-to-SNN Conversioncited by 10
  5. 2024SPikE-SSM: A Sparse, Precise, and Efficient Spiking State Space Model for Long Sequences Learningcited by 9
  6. 2024Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networkscited by 8
  7. 2024Brain-Inspired Online Adaptation for Remote Sensing with Spiking Neural Networkcited by 8
  8. 2024Advancing Training Efficiency of Deep Spiking Neural Networks through Rate-based Backpropagationcited by 8
  9. 2025Efficient Parallel Training Methods for Spiking Neural Networks with Constant Time Complexitycited by 5
  10. 2025Improving Long-term Autoregressive Spatiotemporal Predictions: A Proof of Concept with Fluid Dynamicscited by 4
  11. 2024Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networkscited by 3
  12. 2025SpikingMamba: Towards Energy-Efficient Large Language Models via Knowledge Distillation from Mambacited by 3
25 more
  1. 2025Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deploymentcited by 2
  2. 2025Edge Intelligence with Spiking Neural Networkscited by 2
  3. 2025Dendritic Resonate-and-Fire Neuron for Effective and Efficient Long Sequence Modelingcited by 2
  4. 2025BSO: Binary Spiking Online Optimization Algorithmcited by 2
  5. 2025Efficient Spike-driven Transformer for High-performance Drone-View Geo-Localizationcited by 2
  6. 2024STOP: Spatiotemporal Orthogonal Propagation for Weight-Threshold-Leakage Synergistic Training of Deep Spiking Neural Networkscited by 1
  7. 2025A Truly Sparse and General Implementation of Gradient-Based Synaptic Plasticitycited by 1
  8. 2025Spiking Neural Network Feature Discrimination Boosts Modality Fusioncited by 1
  9. 2025Threshold Modulation for Online Test-Time Adaptation of Spiking Neural Networkscited by 1
  10. 2025Efficient Training of Spiking Neural Networks by Spike-aware Data Pruningcited by 1
  11. 2026RMAAT: Astrocyte-Inspired Memory Compression and Replay for Efficient Long-Context Transformerscited by 1
  12. 2023When Bio-Inspired Computing meets Deep Learning: Low-Latency, Accurate, & Energy-Efficient Spiking Neural Networks from Artificial Neural Networkscited by 0
  13. 2024Topology Optimization of Random Memristors for Input-Aware Dynamic SNNcited by 0
  14. 2024A dynamic vision sensor object recognition model based on trainable event-driven convolution and spiking attention mechanismcited by 0
  15. 2024Rethinking SNN Online Training and Deployment: Gradient-Coherent Learning via Hybrid-Driven LIF Modelcited by 0
  16. 2025ILIF: Temporal Inhibitory Leaky Integrate-and-Fire Neuron for Overactivation in Spiking Neural Networkscited by 0
  17. 2025TDFormer: A Top-Down Attention-Controlled Spiking Transformercited by 0
  18. 2025Optimization of Low-Latency Spiking Neural Networks Utilizing Historical Dynamics of Refractory Periodscited by 0
  19. 2025A Self-Ensemble Inspired Approach for Effective Training of Binary-Weight Spiking Neural Networkscited by 0
  20. 2025SAFA-SNN: Sparsity-Aware On-Device Few-Shot Class-Incremental Learning with Fast-Adaptive Structure of Spiking Neural Networkcited by 0
  21. 2025Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Trainingcited by 0
  22. 2025Stabilizing Direct Training of Spiking Neural Networks: Membrane Potential Initialization and Threshold-robust Surrogate Gradientcited by 0
  23. 2026Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networkscited by 0
  24. 2026NeuroTrain: Surveying Local Learning Rules for Spiking Neural Networks with an Open Benchmarking Frameworkcited by 0
  25. 2026Event-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformerscited by 0