mlsanityData to 7 Oct 2026

Advancing Training Efficiency of Deep Spiking Neural Networks through Rate-based Backpropagation

arXiv 2410.11488 ↗First posted 15 Oct 2024cs.LG cs.NEcited by 8

Not on the map. Why?

Abstract (arXiv)

Recent insights have revealed that rate-coding is a primary form of information representation captured by surrogate-gradient-based Backpropagation Through Time (BPTT) in training deep Spiking Neural Networks (SNNs). Motivated by these findings, we propose rate-based backpropagation, a training strategy specifically designed to exploit rate-based representations to reduce the complexity of BPTT. Our method minimizes reliance on detailed temporal derivatives by focusing on averaged dynamics, streamlining the computational graph to reduce memory and computational demands of SNNs training. We substantiate the rationality of the gradient approximation between BPTT and the proposed method through both theoretical analysis and empirical observations. Comprehensive experiments on CIFAR-10, CIFAR-100, ImageNet, and CIFAR10-DVS validate that our method achieves comparable performance to BPTT counterparts, and surpasses state-of-the-art efficient training techniques. By leveraging the inherent benefits of rate-coding, this work sets the stage for more scalable and efficient SNNs training within resource-constrained environments. Our code is available at https://github.com/Tab-ct/rate-based-backpropagation.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 49 of its 88 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

49 of 88 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  7. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  8. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 604
  9. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 399
  10. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  11. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 340
  12. 2021Deep Residual Learning in Spiking Neural Networkscited by 326
37 more
  1. 2020Going Deeper With Directly-Trained Larger Spiking Neural Networkscited by 268
  2. 2018SLAYER: Spike Layer Error Reassignment in Timecited by 257
  3. 2018Going Deeper in Spiking Neural Networks: VGG and Residual Architecturescited by 236
  4. 2016Training Deep Spiking Neural Networks using Backpropagationcited by 203
  5. 2017SuperSpike: Supervised learning in multi-layer spiking neural networkscited by 178
  6. 2019Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detectioncited by 174
  7. 2023SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligencecited by 163
  8. 2020RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Networkcited by 141
  9. 2021Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networkscited by 134
  10. 2023Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networkscited by 116
  11. 2021A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibrationcited by 111
  12. 2021Temporal-wise Attention Spiking Neural Networks for Event Streams Classificationcited by 106
  13. 2020Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networkscited by 98
  14. 2022Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representationcited by 89
  15. 2016Supervised learning based on temporal coding in spiking neural networkscited by 75
  16. 2021Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networkscited by 74
  17. 2022GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networkscited by 63
  18. 2021HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noisecited by 61
  19. 2020Event-Based Backpropagation can compute Exact Gradients for Spiking Neural Networkscited by 59
  20. 2022Online Training Through Time for Spiking Neural Networkscited by 48
  21. 2020Inherent Adversarial Robustness of Deep Spiking Neural Networks: Effects of Discrete Input Encoding and Non-Linear Activationscited by 47
  22. 2019Temporal-Coded Deep Spiking Neural Network with Easy Training and Robust Performancecited by 45
  23. 2020Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networkscited by 43
  24. 2021Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium Statecited by 39
  25. 2023Reducing ANN-SNN Conversion Error through Residual Membrane Potentialcited by 39
  26. 2023Bridging the Gap between ANNs and SNNs by Calibrating Offset Spikescited by 38
  27. 2023Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networkscited by 37
  28. 2021Accurate online training of dynamical spiking neural networks through Forward Propagation Through Timecited by 33
  29. 2020Online Spatio-Temporal Learning in Deep Neural Networkscited by 32
  30. 2023Direct Learning-Based Deep Spiking Neural Networks: A Reviewcited by 31
  31. 2020Unifying Activation- and Timing-based Learning Rules for Spiking Neural Networkscited by 27
  32. 2020Securing Deep Spiking Neural Networks against Adversarial Attacks through Inherent Structural Parameterscited by 27
  33. 2022STSC-SNN: Spatio-Temporal Synaptic Connection with Temporal Convolution and Attention for Spiking Neural Networkscited by 20
  34. 2019SpikeGrad: An ANN-equivalent Computation Model for Implementing Backpropagation with Spikescited by 15
  35. 2023Uncovering the Representation of Spiking Neural Networks Trained with Surrogate Gradientcited by 11
  36. 2016Motor control by precisely timed spike patternscited by 3
  37. 2023Exploiting High Performance Spiking Neural Networks with Efficient Spiking Patternscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2025Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacementcited by 6
  2. 2025Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deploymentcited by 2
  3. 2025Edge Intelligence with Spiking Neural Networkscited by 2
  4. 2025Optimization of Low-Latency Spiking Neural Networks Utilizing Historical Dynamics of Refractory Periodscited by 0
  5. 2025Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Trainingcited by 0
  6. 2026Stable Spike: Dual Consistency Optimization via Bitwise AND Operations for Spiking Neural Networkscited by 0
  7. 2026Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networkscited by 0
  8. 2026Unified Complex-valued Neural Network: A Magnitude-Phase Computational Model for Event-Driven Neuromorphic Learningcited by 0