# A Truly Sparse and General Implementation of Gradient-Based Synaptic Plasticity

arXiv: https://arxiv.org/abs/2501.11407 · HTML: https://mlsanity.com/paper/2501.11407/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 20 Jan 2025.

## Builds on (most cited first, 22 of 22)

- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [Training Spiking Neural Networks Using Lessons From Deep Learning](https://mlsanity.com/paper/2109.12894/) (2021)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence](https://mlsanity.com/paper/2310.16620/) (2023)
- [The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks](https://mlsanity.com/paper/1910.07407/) (2019)
- [Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of neoHebbian Three-Factor Learning Rules](https://mlsanity.com/paper/1801.05219/) (2018)
- [Event-Based Backpropagation can compute Exact Gradients for Spiking Neural Networks](https://mlsanity.com/paper/2009.08378/) (2020)
- [Fast and energy-efficient neuromorphic deep learning with first-spike times](https://mlsanity.com/paper/1912.11443/) (2019)
- [Sparse Spiking Gradient Descent](https://mlsanity.com/paper/2105.08810/) (2021)
- [Brain-Inspired Learning on Neuromorphic Substrates](https://mlsanity.com/paper/2010.11931/) (2020)
- [Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks](https://mlsanity.com/paper/2302.14311/) (2023)
- [Accurate online training of dynamical spiking neural networks through Forward Propagation Through Time](https://mlsanity.com/paper/2112.11231/) (2021)
- [Online Spatio-Temporal Learning in Deep Neural Networks](https://mlsanity.com/paper/2007.12723/) (2020)
- [Neuromorphic Intermediate Representation: A Unified Instruction Set for Interoperable Brain-Inspired Computing](https://mlsanity.com/paper/2311.14641/) (2023)
- [A Practical Sparse Approximation for Real Time Recurrent Learning](https://mlsanity.com/paper/2006.07232/) (2020)
- [EXODUS: Stable and Efficient Training of Spiking Neural Networks](https://mlsanity.com/paper/2205.10242/) (2022)
- [Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models](https://mlsanity.com/paper/2311.04386/) (2023)
- [Slax: A Composable JAX Library for Rapid and Flexible Prototyping of Spiking Neural Networks](https://mlsanity.com/paper/2404.05807/) (2024)
- [SNNAX -- Spiking Neural Networks in JAX](https://mlsanity.com/paper/2409.02842/) (2024)
- [Prospective and retrospective coding in cortical neurons](https://mlsanity.com/paper/2405.14810/) (2024)
- [Optimizing Automatic Differentiation with Deep Reinforcement Learning](https://mlsanity.com/paper/2406.05027/) (2024)

## Built on by (most cited first, 1 of 1)

- [Learning long range dependencies through time reversal symmetry breaking](https://mlsanity.com/paper/2506.05259/) (2025)

## Abstract (arXiv)

Online synaptic plasticity rules derived from gradient descent achieve high accuracy on a wide range of practical tasks. However, their software implementation often requires tediously hand-derived gradients or using gradient backpropagation which sacrifices the online capability of the rules. In this work, we present a custom automatic differentiation (AD) pipeline for sparse and online implementation of gradient-based synaptic plasticity rules that generalizes to arbitrary neuron models. Our work combines the programming ease of backpropagation-type methods for forward AD while being memory-efficient. To achieve this, we exploit the advantageous compute and memory scaling of online synaptic plasticity by providing an inherently sparse implementation of AD where expensive tensor contractions are replaced with simple element-wise multiplications if the tensors are diagonal. Gradient-based synaptic plasticity rules such as eligibility propagation (e-prop) have exactly this property and thus profit immensely from this feature. We demonstrate the alignment of our gradients with respect to gradient backpropagation on an synthetic task where e-prop gradients are exact, as well as audio speech classification benchmarks. We demonstrate how memory utilization scales with network size without dependence on the sequence length, as expected from forward AD methods.