mlsanityData to 7 Oct 2026

Towards Zero Memory Footprint Spiking Neural Network Training

arXiv 2308.08649 ↗First posted 16 Aug 2023cs.NE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Biologically-inspired Spiking Neural Networks (SNNs), processing information using discrete-time events known as spikes rather than continuous values, have garnered significant attention due to their hardware-friendly and energy-efficient characteristics. However, the training of SNNs necessitates a considerably large memory footprint, given the additional storage requirements for spikes or events, leading to a complex structure and dynamic setup. In this paper, to address memory constraint in SNN training, we introduce an innovative framework, characterized by a remarkably low memory footprint. We (i) design a reversible SNN node that retains a high level of accuracy. Our design is able to achieve a 58.65× reduction in memory usage compared to the current SNN node. We (ii) propose a unique algorithm to streamline the backpropagation process of our reversible SNN node. This significantly trims the backward Floating Point Operations Per Second (FLOPs), thereby accelerating the training process in comparison to current reversible layer backpropagation method. By using our algorithm, the training time is able to be curtailed by 23.8% relative to existing reversible layer architectures.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 16 Aug 2023.

Builds on

20 of 33 references.

References · 20

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2016Density estimation using Real NVPcited by 2,029
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  5. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 604
  6. 2021Deep Residual Learning in Spiking Neural Networkscited by 326
  7. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 215
  8. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  9. 2020Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagationcited by 142
  10. 2022Online Training Through Time for Spiking Neural Networkscited by 48
  11. 2017STDP Based Pruning of Connections and Weight Quantization in Spiking Neural Networks for Energy Efficient Recognitioncited by 41
  12. 2021Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium Statecited by 39
8 more
  1. 2023Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorizationcited by 34
  2. 2021Q-SpiNN: A Framework for Quantizing Spiking Neural Networkscited by 33
  3. 2021WINNet: Wavelet-inspired Invertible Network for Image Denoisingcited by 33
  4. 2023Reversible Vision Transformerscited by 33
  5. 2018Reversible Recurrent Neural Networkscited by 22
  6. 2023Neurogenesis Dynamics-inspired Spiking Neural Network Training Accelerationcited by 10
  7. 2022SNN2ANN: A Fast and Memory-Efficient Training Framework for Spiking Neural Networkscited by 6
  8. 2021Memory Efficient 3D U-Net with Reversible Mobile Inverted Bottlenecks for Brain Tumor Segmentationcited by 3

Built on by

Not cited yet.