mlsanityData to 14 Oct 2026

Sparse Spiking Gradient Descent

arXiv 2105.08810 ↗First posted 18 May 2021cs.NE cs.ET cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

There is an increasing interest in emulating Spiking Neural Networks (SNNs) on neuromorphic computing devices due to their low energy consumption. Recent advances have allowed training SNNs to a point where they start to compete with traditional Artificial Neural Networks (ANNs) in terms of accuracy, while at the same time being energy efficient when run on neuromorphic hardware. However, the process of training SNNs is still based on dense tensor operations originally developed for ANNs which do not leverage the spatiotemporally sparse nature of SNNs. We present here the first sparse SNN backpropagation algorithm which achieves the same or better accuracy as current state of the art methods while being significantly faster and more memory efficient. We show the effectiveness of our method on real datasets of varying complexity (Fashion-MNIST, Neuromophic-MNIST and Spiking Heidelberg Digits) achieving a speedup in the backward pass of up to 150x, and 85% more memory efficient, without losing accuracy.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 22 of its 62 references that are in the map. First posted on arXiv 18 May 2021.

Builds on

22 of 62 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  4. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  5. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 626
  6. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  7. 2015Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccadescited by 314
  8. 2018SLAYER: Spike Layer Error Reassignment in Timecited by 266
  9. 2019The Heidelberg spiking datasets for the systematic evaluation of spiking neural networkscited by 137
  10. 2016Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computingcited by 70
  11. 2018Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of neoHebbian Three-Factor Learning Rulescited by 67
  12. 2015Spiking Deep Networks with LIF Neuronscited by 63
10 more
  1. 2020Event-Based Backpropagation can compute Exact Gradients for Spiking Neural Networkscited by 59
  2. 2015Memory and information processing in neuromorphic systemscited by 48
  3. 2019SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systemscited by 46
  4. 2020Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networkscited by 43
  5. 2021Pruning of Deep Spiking Neural Networks through Gradient Rewiringcited by 43
  6. 2016Conversion of Artificial Recurrent Neural Networks to Spiking Neural Networks for Low-power Neuromorphic Hardwarecited by 40
  7. 2017Adaptive Sampled Softmax with Kernel Based Samplingcited by 38
  8. 2014Neuromorphic electronic circuits for building autonomous cognitive systemscited by 19
  9. 2017Neuromorphic Hardware In The Loop: Training a Deep Spiking Network on the BrainScaleS Wafer-Scale Systemcited by 19
  10. 2021Accelerating SLIDE Deep Learning on Modern CPUs: Vectorization, Quantizations, Memory Optimizations, and Morecited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 12; 2024: 10; 2025: 7; 2026: 2.

41 papers.

Sort:

  1. 2021Training Spiking Neural Networks Using Lessons From Deep Learningcited by 259
  2. 2021Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networkscited by 99
  3. 2023Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependenciescited by 41
  4. 2023Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networkscited by 37
  5. 2024Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticketcited by 30
  6. 2022Efficient recurrent architectures through activity sparsity and sparse back-propagation through timecited by 26
  7. 2022Loss shaping enhances exact gradient learning with Eventprop in spiking neural networkscited by 14
  8. 2024Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithmscited by 12
  9. 2025Temporal-Guided Spiking Neural Networks for Event-Based Human Action Recognitioncited by 11
  10. 2022Spiking neural networks for nonlinear regressioncited by 10
  11. 2024SpGesture: Source-Free Domain-adaptive sEMG-based Gesture Recognition with Jaccard Attentive Spiking Neural Networkcited by 9
  12. 2022The fine line between dead neurons and sparsity in binarized spiking neural networkscited by 8
29 more
  1. 2023A Comprehensive Review of Spiking Neural Networks: Interpretation, Optimization, Efficiency, and Best Practicescited by 8
  2. 2021Learning through structure: towards deep neuromorphic knowledge graph embeddingscited by 7
  3. 2022SNN2ANN: A Fast and Memory-Efficient Training Framework for Spiking Neural Networkscited by 7
  4. 2023Learning Bottleneck Transformer for Event Image-Voxel Feature Fusion based Classificationcited by 6
  5. 2024Zero-Shot Temporal Resolution Domain Adaptation for Spiking Neural Networkscited by 6
  6. 2025Backpropagation-free Spiking Neural Networks with the Forward-Forward Algorithmcited by 5
  7. 2022EXODUS: Stable and Efficient Training of Spiking Neural Networkscited by 4
  8. 2023Point-Voxel Absorbing Graph Representation Learning for Event Stream based Recognitioncited by 4
  9. 2023Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Modelscited by 4
  10. 2023Addressing the speed-accuracy simulation trade-off for adaptive spiking neuronscited by 4
  11. 2022Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rulescited by 3
  12. 2024Retain, Blend, and Exchange: A Quality-aware Spatial-Stereo Fusion Approach for Event Stream Recognitioncited by 3
  13. 2025Beyond Rate Coding: Surrogate Gradients Enable Spike Timing Learning in Spiking Neural Networkscited by 3
  14. 2026Parallelizable memory recurrent unitscited by 3
  15. 2023Spiking Network Initialisation and Firing Rate Collapsecited by 2
  16. 2024CLASSP: a Biologically-Inspired Approach to Continual Learning through Adjustment Suppression and Sparsity Promotioncited by 2
  17. 2025Edge Intelligence with Spiking Neural Networkscited by 2
  18. 2023Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Trainingcited by 1
  19. 2024Uncertainty-aware Bridge based Mobile-Former Network for Event-based Pattern Recognitioncited by 1
  20. 2025A Truly Sparse and General Implementation of Gradient-Based Synaptic Plasticitycited by 1
  21. 2025Threshold Adaptation in Spiking Networks Enables Shortest Path Finding and Place Disambiguationcited by 1
  22. 2023Energy Efficient Training of SNN using Local Zeroth Order Methodcited by 0
  23. 2023Membrane Potential Distribution Adjustment and Parametric Surrogate Gradient in Spiking Neural Networkscited by 0
  24. 2023Neuromorphic Online Learning for Spatiotemporal Patterns with a Forward-only Timelinecited by 0
  25. 2024STAL: Spike Threshold Adaptive Learning Encoder for Classification of Pain-Related Biosignal Datacited by 0
  26. 2024Iteration over event space in time-to-first-spike spiking neural networks for Twitter bot classificationcited by 0
  27. 2024Synaptic Modulation using Interspike Intervals Increases Energy Efficiency of Spiking Neural Networkscited by 0
  28. 2025Learning with Spike Synchrony in Spiking Neural Networkscited by 0
  29. 2026SAGE: Surrogate-gradient Adaptation via Attention-Guided Entropy for Spiking Transformerscited by 0