# Spiking Network Initialisation and Firing Rate Collapse

arXiv: https://arxiv.org/abs/2305.08879 · HTML: https://mlsanity.com/paper/2305.08879/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 29 references that are in the map.
- First posted on arXiv 13 May 2023.

## Builds on (most cited first, 10 of 10)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)
- [Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks](https://mlsanity.com/paper/1806.05393/) (2018)
- [Sparse Spiking Gradient Descent](https://mlsanity.com/paper/2105.08810/) (2021)
- [How to Initialize your Network? Robust Initialization for WeightNorm & ResNets](https://mlsanity.com/paper/1906.02341/) (2019)
- [Fluctuation-driven initialization for spiking neural network training](https://mlsanity.com/paper/2206.10226/) (2022)
- [Revise Saturated Activation Functions](https://mlsanity.com/paper/1602.05980/) (2016)

## Built on by (most cited first, 2 of 2)

- [Addressing the speed-accuracy simulation trade-off for adaptive spiking neurons](https://mlsanity.com/paper/2311.11390/) (2023)
- [Deep activity propagation via weight initialization in spiking neural networks](https://mlsanity.com/paper/2410.00580/) (2024)

## Abstract (arXiv)

In recent years, newly developed methods to train spiking neural networks (SNNs) have rendered them as a plausible alternative to Artificial Neural Networks (ANNs) in terms of accuracy, while at the same time being much more energy efficient at inference and potentially at training time. However, it is still unclear what constitutes a good initialisation for an SNN. We often use initialisation schemes developed for ANN training which are often inadequate and require manual tuning. In this paper, we attempt to tackle this issue by using techniques from the ANN initialisation literature as well as computational neuroscience results. We show that the problem of weight initialisation for ANNs is a more nuanced problem than it is for ANNs due to the spike-and-reset non-linearity of SNNs and the firing rate collapse problem. We firstly identify and propose several solutions to the firing rate collapse problem under different sets of assumptions which successfully solve the issue by leveraging classical random walk and Wiener processes results. Secondly, we devise a general strategy for SNN initialisation which combines variance propagation techniques from ANNs and different methods to obtain the expected firing rate and membrane potential distribution based on diffusion and shot-noise approximations. Altogether, we obtain theoretical results to solve the SNN initialisation which consider the membrane potential distribution in the presence of a threshold. Yet, to what extent can these methods be successfully applied to SNNs on real datasets remains an open question.