# Training Deep Spiking Auto-encoders without Bursting or Dying Neurons through Regularization

arXiv: https://arxiv.org/abs/2109.11045 · HTML: https://mlsanity.com/paper/2109.11045/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 22 Sep 2021.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades](https://mlsanity.com/paper/1507.07629/) (2015)
- [Training Deep Spiking Neural Networks using Backpropagation](https://mlsanity.com/paper/1608.08782/) (2016)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures](https://mlsanity.com/paper/1903.06379/) (2019)
- [A Survey of Neuromorphic Computing and Neural Networks in Hardware](https://mlsanity.com/paper/1705.06963/) (2017)
- [How Can We Be So Dense? The Benefits of Using Highly Sparse Representations](https://mlsanity.com/paper/1903.11257/) (2019)
- [Collapse of Deep and Narrow Neural Nets](https://mlsanity.com/paper/1808.04947/) (2018)
- [Biologically plausible deep learning -- but how far can we go with shallow networks?](https://mlsanity.com/paper/1905.04101/) (2019)
- [Multimodal VAE Active Inference Controller](https://mlsanity.com/paper/2103.04412/) (2021)
- [Training Spiking Deep Networks for Neuromorphic Hardware](https://mlsanity.com/paper/1611.05141/) (2016)
- [Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization](https://mlsanity.com/paper/1812.00481/) (2018)
- [Bio-Inspired Spiking Convolutional Neural Network using Layer-wise Sparse Coding and STDP Learning](https://mlsanity.com/paper/1611.03000/) (2016)
- [Temporally Efficient Deep Learning with Spikes](https://mlsanity.com/paper/1706.04159/) (2017)
- [Unsupervised Regenerative Learning of Hierarchical Features in Spiking Deep Networks for Object Recognition](https://mlsanity.com/paper/1602.01510/) (2016)
- [Deep Spiking Networks](https://mlsanity.com/paper/1602.08323/) (2016)
- [Synthesizing Images from Spatio-Temporal Representations using Spike-based Backpropagation](https://mlsanity.com/paper/1906.08861/) (2019)

## Built on by (most cited first, 2 of 2)

- [Closed-form control with spike coding networks](https://mlsanity.com/paper/2212.12887/) (2022)
- [Spiking Neural Networks for Continuous Control via End-to-End Model-Based Learning](https://mlsanity.com/paper/2509.05356/) (2025)

## Abstract (arXiv)

Spiking neural networks are a promising approach towards next-generation models of the brain in computational neuroscience. Moreover, compared to classic artificial neural networks, they could serve as an energy-efficient deployment of AI by enabling fast computation in specialized neuromorphic hardware. However, training deep spiking neural networks, especially in an unsupervised manner, is challenging and the performance of a spiking model is significantly hindered by dead or bursting neurons. Here, we apply end-to-end learning with membrane potential-based backpropagation to a spiking convolutional auto-encoder with multiple trainable layers of leaky integrate-and-fire neurons. We propose bio-inspired regularization methods to control the spike density in latent representations. In the experiments, we show that applying regularization on membrane potential and spiking output successfully avoids both dead and bursting neurons and significantly decreases the reconstruction error of the spiking auto-encoder. Training regularized networks on the MNIST dataset yields image reconstruction quality comparable to non-spiking baseline models (deterministic and variational auto-encoder) and indicates improvement upon earlier approaches. Importantly, we show that, unlike the variational auto-encoder, the spiking latent representations display structure associated with the image class.