# Sparse Spiking Gradient Descent

arXiv: https://arxiv.org/abs/2105.08810 · HTML: https://mlsanity.com/paper/2105.08810/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 62 references that are in the map.
- First posted on arXiv 18 May 2021.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades](https://mlsanity.com/paper/1507.07629/) (2015)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks](https://mlsanity.com/paper/1910.07407/) (2019)
- [Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing](https://mlsanity.com/paper/1603.08270/) (2016)
- [Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of neoHebbian Three-Factor Learning Rules](https://mlsanity.com/paper/1801.05219/) (2018)
- [Spiking Deep Networks with LIF Neurons](https://mlsanity.com/paper/1510.08829/) (2015)
- [Event-Based Backpropagation can compute Exact Gradients for Spiking Neural Networks](https://mlsanity.com/paper/2009.08378/) (2020)
- [Memory and information processing in neuromorphic systems](https://mlsanity.com/paper/1506.03264/) (2015)
- [SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems](https://mlsanity.com/paper/1903.03129/) (2019)
- [Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks](https://mlsanity.com/paper/2005.11633/) (2020)
- [Pruning of Deep Spiking Neural Networks through Gradient Rewiring](https://mlsanity.com/paper/2105.04916/) (2021)
- [Conversion of Artificial Recurrent Neural Networks to Spiking Neural Networks for Low-power Neuromorphic Hardware](https://mlsanity.com/paper/1601.04187/) (2016)
- [Adaptive Sampled Softmax with Kernel Based Sampling](https://mlsanity.com/paper/1712.00527/) (2017)
- [Neuromorphic electronic circuits for building autonomous cognitive systems](https://mlsanity.com/paper/1403.6428/) (2014)
- [Neuromorphic Hardware In The Loop: Training a Deep Spiking Network on the BrainScaleS Wafer-Scale System](https://mlsanity.com/paper/1703.01909/) (2017)
- [Accelerating SLIDE Deep Learning on Modern CPUs: Vectorization, Quantizations, Memory Optimizations, and More](https://mlsanity.com/paper/2103.10891/) (2021)

## Built on by (most cited first, 30 of 41)

- [Training Spiking Neural Networks Using Lessons From Deep Learning](https://mlsanity.com/paper/2109.12894/) (2021)
- [Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks](https://mlsanity.com/paper/2106.01862/) (2021)
- [Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependencies](https://mlsanity.com/paper/2304.12760/) (2023)
- [Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks](https://mlsanity.com/paper/2302.14311/) (2023)
- [Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticket](https://mlsanity.com/paper/2401.02020/) (2024)
- [Efficient recurrent architectures through activity sparsity and sparse back-propagation through time](https://mlsanity.com/paper/2206.06178/) (2022)
- [Loss shaping enhances exact gradient learning with Eventprop in spiking neural networks](https://mlsanity.com/paper/2212.01232/) (2022)
- [Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithms](https://mlsanity.com/paper/2408.09764/) (2024)
- [Temporal-Guided Spiking Neural Networks for Event-Based Human Action Recognition](https://mlsanity.com/paper/2503.17132/) (2025)
- [Spiking neural networks for nonlinear regression](https://mlsanity.com/paper/2210.03515/) (2022)
- [SpGesture: Source-Free Domain-adaptive sEMG-based Gesture Recognition with Jaccard Attentive Spiking Neural Network](https://mlsanity.com/paper/2405.14398/) (2024)
- [The fine line between dead neurons and sparsity in binarized spiking neural networks](https://mlsanity.com/paper/2201.11915/) (2022)
- [A Comprehensive Review of Spiking Neural Networks: Interpretation, Optimization, Efficiency, and Best Practices](https://mlsanity.com/paper/2303.10780/) (2023)
- [Learning through structure: towards deep neuromorphic knowledge graph embeddings](https://mlsanity.com/paper/2109.10376/) (2021)
- [SNN2ANN: A Fast and Memory-Efficient Training Framework for Spiking Neural Networks](https://mlsanity.com/paper/2206.09449/) (2022)
- [Learning Bottleneck Transformer for Event Image-Voxel Feature Fusion based Classification](https://mlsanity.com/paper/2308.11937/) (2023)
- [Zero-Shot Temporal Resolution Domain Adaptation for Spiking Neural Networks](https://mlsanity.com/paper/2411.04760/) (2024)
- [Backpropagation-free Spiking Neural Networks with the Forward-Forward Algorithm](https://mlsanity.com/paper/2502.20411/) (2025)
- [EXODUS: Stable and Efficient Training of Spiking Neural Networks](https://mlsanity.com/paper/2205.10242/) (2022)
- [Point-Voxel Absorbing Graph Representation Learning for Event Stream based Recognition](https://mlsanity.com/paper/2306.05239/) (2023)
- [Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models](https://mlsanity.com/paper/2311.04386/) (2023)
- [Addressing the speed-accuracy simulation trade-off for adaptive spiking neurons](https://mlsanity.com/paper/2311.11390/) (2023)
- [Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules](https://mlsanity.com/paper/2206.00823/) (2022)
- [Retain, Blend, and Exchange: A Quality-aware Spatial-Stereo Fusion Approach for Event Stream Recognition](https://mlsanity.com/paper/2406.18845/) (2024)
- [Beyond Rate Coding: Surrogate Gradients Enable Spike Timing Learning in Spiking Neural Networks](https://mlsanity.com/paper/2507.16043/) (2025)
- [Parallelizable memory recurrent units](https://mlsanity.com/paper/2601.09495/) (2026)
- [Spiking Network Initialisation and Firing Rate Collapse](https://mlsanity.com/paper/2305.08879/) (2023)
- [CLASSP: a Biologically-Inspired Approach to Continual Learning through Adjustment Suppression and Sparsity Promotion](https://mlsanity.com/paper/2405.09637/) (2024)
- [Edge Intelligence with Spiking Neural Networks](https://mlsanity.com/paper/2507.14069/) (2025)
- [Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training](https://mlsanity.com/paper/2312.09391/) (2023)

## Abstract (arXiv)

There is an increasing interest in emulating Spiking Neural Networks (SNNs) on neuromorphic computing devices due to their low energy consumption. Recent advances have allowed training SNNs to a point where they start to compete with traditional Artificial Neural Networks (ANNs) in terms of accuracy, while at the same time being energy efficient when run on neuromorphic hardware. However, the process of training SNNs is still based on dense tensor operations originally developed for ANNs which do not leverage the spatiotemporally sparse nature of SNNs. We present here the first sparse SNN backpropagation algorithm which achieves the same or better accuracy as current state of the art methods while being significantly faster and more memory efficient. We show the effectiveness of our method on real datasets of varying complexity (Fashion-MNIST, Neuromophic-MNIST and Spiking Heidelberg Digits) achieving a speedup in the backward pass of up to 150x, and 85% more memory efficient, without losing accuracy.