mlsanityData to 7 Oct 2026

Rethinking SNN Online Training and Deployment: Gradient-Coherent Learning via Hybrid-Driven LIF Model

arXiv 2410.07547 ↗First posted 10 Oct 2024cs.NE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Spiking Neural Networks (SNNs) are considered to have enormous potential in the future development of Artificial Intelligence due to their brain-inspired and energy-efficient properties. Compared to vanilla Spatial-Temporal Back-propagation (STBP) training methods, online training can effectively avoid the risk of GPU memory explosion. However, current online learning frameworks cannot tackle the gradient discrepancy problem between the forward and backward process, merely aiming to optimize the GPU memory, resulting in no performance advantages compared to the STBP-based models in the inference stage. To address the aforementioned challenges, we propose Hybrid-Driven Leaky Integrate-and-Fire (HD-LIF) model family for efficient online learning, which respectively adopt different spiking calculation mechanism in the upper-region and lower-region of the firing threshold. We theoretically point out that our learning framework can effectively separate temporal gradients and address the misalignment problem of surrogate gradients, as well as achieving full-stage optimization towards learning precision, memory complexity and power consumption. Experimental results have demonstrated that our scheme is enable to achieve state-of-the-art performance for multiple evaluation metrics, breaking through the traditional paradigm of SNN online training and deployment. Code is available at hrefhttps://github.com/hzc1208/HDLIFhere.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 58 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

35 of 58 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2017Squeeze-and-Excitation Networkscited by 4,763
  6. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  7. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 604
  8. 2019ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networkscited by 572
  9. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 399
  10. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 340
  11. 2020Going Deeper With Directly-Trained Larger Spiking Neural Networkscited by 268
  12. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 263
23 more
  1. 2016Ternary Weight Networkscited by 195
  2. 2022Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingcited by 192
  3. 2019Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detectioncited by 174
  4. 2022Attention Spiking Neural Networkscited by 103
  5. 2020Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networkscited by 98
  6. 2024Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chipscited by 97
  7. 2021Advancing Spiking Neural Networks towards Deep Residual Learningcited by 76
  8. 2022GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networkscited by 63
  9. 2020LIAF-Net: Leaky Integrate and Analog Fire Network for Lightweight and Efficient Spatiotemporal Information Processingcited by 56
  10. 2022Online Training Through Time for Spiking Neural Networkscited by 48
  11. 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46
  12. 2024SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networkscited by 38
  13. 2023Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networkscited by 37
  14. 2023Membrane Potential Batch Normalization for Spiking Neural Networkscited by 35
  15. 2021Accurate online training of dynamical spiking neural networks through Forward Propagation Through Timecited by 33
  16. 2020Unifying Activation- and Timing-based Learning Rules for Spiking Neural Networkscited by 27
  17. 2023Gated Attention Coding for Training High-performance and Efficient Spiking Neural Networkscited by 26
  18. 2022Training Spiking Neural Networks with Local Tandem Learningcited by 22
  19. 2024CLIF: Complementary Leaky Integrate-and-Fire Neuron for Spiking Neural Networkscited by 21
  20. 2024High-Performance Temporal Reversible Spiking Neural Networks with O(L) Training Memory and O(1) Inference Costcited by 14
  21. 2022Join the High Accuracy Club on ImageNet with A Binary Neural Network Ticketcited by 13
  22. 2024Autaptic Synaptic Circuit Enhances Spatio-temporal Predictive Learning of Spiking Neural Networkscited by 8
  23. 2023Memory-Efficient Reversible Spiking Neural Networkscited by 6

Built on by

Not cited yet.