# Towards Zero Memory Footprint Spiking Neural Network Training

arXiv: https://arxiv.org/abs/2308.08649 · HTML: https://mlsanity.com/paper/2308.08649/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 16 Aug 2023.

## Builds on (most cited first, 20 of 20)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Deep Residual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2102.04159/) (2021)
- [The Reversible Residual Network: Backpropagation Without Storing Activations](https://mlsanity.com/paper/1707.04585/) (2017)
- [Training Graph Neural Networks with 1000 Layers](https://mlsanity.com/paper/2106.07476/) (2021)
- [Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation](https://mlsanity.com/paper/2005.01807/) (2020)
- [Online Training Through Time for Spiking Neural Networks](https://mlsanity.com/paper/2210.04195/) (2022)
- [STDP Based Pruning of Connections and Weight Quantization in Spiking Neural Networks for Energy Efficient Recognition](https://mlsanity.com/paper/1710.04734/) (2017)
- [Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State](https://mlsanity.com/paper/2109.14247/) (2021)
- [Q-SpiNN: A Framework for Quantizing Spiking Neural Networks](https://mlsanity.com/paper/2107.01807/) (2021)
- [Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorization](https://mlsanity.com/paper/2305.02099/) (2023)
- [WINNet: Wavelet-inspired Invertible Network for Image Denoising](https://mlsanity.com/paper/2109.06381/) (2021)
- [Reversible Vision Transformers](https://mlsanity.com/paper/2302.04869/) (2023)
- [Reversible Recurrent Neural Networks](https://mlsanity.com/paper/1810.10999/) (2018)
- [Neurogenesis Dynamics-inspired Spiking Neural Network Training Acceleration](https://mlsanity.com/paper/2304.12214/) (2023)
- [SNN2ANN: A Fast and Memory-Efficient Training Framework for Spiking Neural Networks](https://mlsanity.com/paper/2206.09449/) (2022)
- [Memory Efficient 3D U-Net with Reversible Mobile Inverted Bottlenecks for Brain Tumor Segmentation](https://mlsanity.com/paper/2104.09648/) (2021)

## Abstract (arXiv)

Biologically-inspired Spiking Neural Networks (SNNs), processing information using discrete-time events known as spikes rather than continuous values, have garnered significant attention due to their hardware-friendly and energy-efficient characteristics. However, the training of SNNs necessitates a considerably large memory footprint, given the additional storage requirements for spikes or events, leading to a complex structure and dynamic setup. In this paper, to address memory constraint in SNN training, we introduce an innovative framework, characterized by a remarkably low memory footprint. We (i) design a reversible SNN node that retains a high level of accuracy. Our design is able to achieve a 58.65× reduction in memory usage compared to the current SNN node. We (ii) propose a unique algorithm to streamline the backpropagation process of our reversible SNN node. This significantly trims the backward Floating Point Operations Per Second (FLOPs), thereby accelerating the training process in comparison to current reversible layer backpropagation method. By using our algorithm, the training time is able to be curtailed by 23.8% relative to existing reversible layer architectures.