# The fine line between dead neurons and sparsity in binarized spiking neural networks

arXiv: https://arxiv.org/abs/2201.11915 · HTML: https://mlsanity.com/paper/2201.11915/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 45 references that are in the map.
- First posted on arXiv 28 Jan 2022.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)
- [The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks](https://mlsanity.com/paper/1910.07407/) (2019)
- [Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1](https://mlsanity.com/paper/1602.02830/) (2016)
- [Efficient Neuromorphic Signal Processing with Loihi 2](https://mlsanity.com/paper/2111.03746/) (2021)
- [Sparse Spiking Gradient Descent](https://mlsanity.com/paper/2105.08810/) (2021)
- [Exploring the Connection Between Binary and Spiking Neural Networks](https://mlsanity.com/paper/2002.10064/) (2020)
- [Hardware Implementation of Deep Network Accelerators Towards Healthcare and Biomedical Applications](https://mlsanity.com/paper/2007.05657/) (2020)
- [Synaptic metaplasticity in binarized neural networks](https://mlsanity.com/paper/2101.07592/) (2021)
- [Learning Memory Access Patterns](https://mlsanity.com/paper/1803.02329/) (2018)
- [BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learning](https://mlsanity.com/paper/2007.04039/) (2020)
- [A Spike in Performance: Training Hybrid-Spiking Neural Networks with Quantized Activation Functions](https://mlsanity.com/paper/2002.03553/) (2020)

## Built on by (most cited first, 8 of 8)

- [SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks](https://mlsanity.com/paper/2302.13939/) (2023)
- [Learning Long Sequences in Spiking Neural Networks](https://mlsanity.com/paper/2401.00955/) (2023)
- [Efficient Deployment of Spiking Neural Networks on SpiNNaker2 for DVS Gesture Recognition Using Neuromorphic Intermediate Representation](https://mlsanity.com/paper/2504.06748/) (2025)
- [Navigating Local Minima in Quantized Spiking Neural Networks](https://mlsanity.com/paper/2202.07221/) (2022)
- [Low Precision Quantization-aware Training in Spiking Neural Networks with Differentiable Quantization Function](https://mlsanity.com/paper/2305.19295/) (2023)
- [SpikeFit: Towards Optimal Deployment of Spiking Networks on Neuromorphic Hardware](https://mlsanity.com/paper/2510.15542/) (2025)
- [A Fully Memristive Spiking Neural Network with Unsupervised Learning](https://mlsanity.com/paper/2203.01416/) (2022)
- [The Hardware Impact of Quantization and Pruning for Weights in Spiking Neural Networks](https://mlsanity.com/paper/2302.04174/) (2023)

## Abstract (arXiv)

Spiking neural networks can compensate for quantization error by encoding information either in the temporal domain, or by processing discretized quantities in hidden states of higher precision. In theory, a wide dynamic range state-space enables multiple binarized inputs to be accumulated together, thus improving the representational capacity of individual neurons. This may be achieved by increasing the firing threshold, but make it too high and sparse spike activity turns into no spike emission. In this paper, we propose the use of `threshold annealing' as a warm-up method for firing thresholds. We show it enables the propagation of spikes across multiple layers where neurons would otherwise cease to fire, and in doing so, achieve highly competitive results on four diverse datasets, despite using binarized weights. Source code is available at https://github.com/jeshraghian/snn-tha/