# High-performance deep spiking neural networks with 0.3 spikes per neuron

arXiv: https://arxiv.org/abs/2306.08744 · HTML: https://mlsanity.com/paper/2306.08744/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 83 references that are in the map.
- First posted on arXiv 14 Jun 2023.

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink](https://mlsanity.com/paper/2204.05149/) (2022)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)
- [Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks](https://mlsanity.com/paper/2303.04347/) (2023)
- [Convergence of Edge Computing and Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/1907.08349/) (2019)
- [A Study of Face Obfuscation in ImageNet](https://mlsanity.com/paper/2103.06191/) (2021)
- [Supervised learning based on temporal coding in spiking neural networks](https://mlsanity.com/paper/1606.08165/) (2016)
- [Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Bounds](https://mlsanity.com/paper/2111.01395/) (2021)
- [Event-Based Backpropagation can compute Exact Gradients for Spiking Neural Networks](https://mlsanity.com/paper/2009.08378/) (2020)
- [Fast and energy-efficient neuromorphic deep learning with first-spike times](https://mlsanity.com/paper/1912.11443/) (2019)
- [Gradient Descent for Spiking Neural Networks](https://mlsanity.com/paper/1706.04698/) (2017)
- [Temporal-Coded Deep Spiking Neural Network with Easy Training and Robust Performance](https://mlsanity.com/paper/1909.10837/) (2019)
- [Temporal Coding in Spiking Neural Networks with Alpha Synaptic Function: Learning with Backpropagation](https://mlsanity.com/paper/1907.13223/) (2019)
- [Conversion of Artificial Recurrent Neural Networks to Spiking Neural Networks for Low-power Neuromorphic Hardware](https://mlsanity.com/paper/1601.04187/) (2016)
- [STDP-based spiking deep convolutional neural networks for object recognition](https://mlsanity.com/paper/1611.01421/) (2016)
- [Google Neural Network Models for Edge Devices: Analyzing and Mitigating Machine Learning Inference Bottlenecks](https://mlsanity.com/paper/2109.14320/) (2021)
- [Neuromorphic Hardware In The Loop: Training a Deep Spiking Network on the BrainScaleS Wafer-Scale System](https://mlsanity.com/paper/1703.01909/) (2017)
- [Demonstrating Advantages of Neuromorphic Computation: A Pilot Study](https://mlsanity.com/paper/1811.03618/) (2018)
- [Random Walk Initialization for Training Very Deep Feedforward Networks](https://mlsanity.com/paper/1412.6558/) (2014)
- [An Exact Mapping From ReLU Networks to Spiking Neural Networks](https://mlsanity.com/paper/2212.12522/) (2022)
- [Improving Noise Tolerance of Mixed-Signal Neural Networks](https://mlsanity.com/paper/1904.01705/) (2019)
- [On the choice of metric in gradient-based theories of brain function](https://mlsanity.com/paper/1805.11851/) (2018)

## Built on by (most cited first, 22 of 22)

- [DelGrad: Exact event-based gradients for training delays and weights on spiking neuromorphic hardware](https://mlsanity.com/paper/2404.19165/) (2024)
- [Stable Learning Using Spiking Neural Networks Equipped With Affine Encoders and Decoders](https://mlsanity.com/paper/2404.04549/) (2024)
- [Time to Spike? Understanding the Representational Power of Spiking Neural Networks in Discrete Time](https://mlsanity.com/paper/2505.18023/) (2025)
- [Deep Oscillatory Neural Network](https://mlsanity.com/paper/2405.03725/) (2024)
- [Causal pieces: analysing and improving spiking neural networks piece by piece](https://mlsanity.com/paper/2504.14015/) (2025)
- [Spiking Music: Audio Compression with Event Based Auto-encoders](https://mlsanity.com/paper/2402.01571/) (2024)
- [Sparse by Rule: Probability-Based N:M Pruning for Spiking Neural Networks](https://mlsanity.com/paper/2511.12097/) (2025)
- [Orthogonal Weight Modification Enhances Learning Scalability and Convergence Efficiency without Gradient Backpropagation](https://mlsanity.com/paper/2602.22259/) (2026)
- [Equivalence of approximation by networks of single- and multi-spike neurons](https://mlsanity.com/paper/2603.13478/) (2026)
- [Vision SmolMamba: Spike-Guided Token Pruning for Energy-Efficient Spiking State-Space Vision Models](https://mlsanity.com/paper/2604.25570/) (2026)
- [Revisiting Reset Mechanisms in Spiking Neural Networks for Sequential Modeling: Specialized Discretization for Binary Activated RNN](https://mlsanity.com/paper/2504.17751/) (2025)
- [Frozen Backpropagation: Relaxing Weight Symmetry in Deep Spiking Neural Networks](https://mlsanity.com/paper/2505.13741/) (2025)
- [Adaptive Time-step Training for Enhancing Spike-Based Neural Radiance Fields](https://mlsanity.com/paper/2507.23033/) (2025)
- [Complementary Learning System Empowers Online Continual Learning of Vehicle Motion Forecasting in Smart Cities](https://mlsanity.com/paper/2508.19597/) (2025)
- [Predictive Spike Timing Enables Distributed Shortest Path Computation in Spiking Neural Networks](https://mlsanity.com/paper/2509.10077/) (2025)
- [NEXUS: Bit-Exact ANN-to-SNN Equivalence via Neuromorphic Gate Circuits with Surrogate-Free Training](https://mlsanity.com/paper/2601.21279/) (2026)
- [Spark: Modular Spiking Neural Networks](https://mlsanity.com/paper/2602.02306/) (2026)
- [Wi-Spike: A Low-power WiFi Human Multi-action Recognition Model with Spiking Neural Networks](https://mlsanity.com/paper/2603.14475/) (2026)
- [Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks](https://mlsanity.com/paper/2607.08406/) (2026)
- [Scalable Training of Continuous-Time Spiking Neural Networks with Differentiable Spike-Time Discretization](https://mlsanity.com/paper/2607.14672/) (2026)
- [SAGE: Surrogate-gradient Adaptation via Attention-Guided Entropy for Spiking Transformers](https://mlsanity.com/paper/2608.13702/) (2026)
- [Large Language Models with At Most One Spike per Neuron](https://mlsanity.com/paper/2609.05151/) (2026)

## Abstract (arXiv)

Communication by rare, binary spikes is a key factor for the energy efficiency of biological brains. However, it is harder to train biologically-inspired spiking neural networks (SNNs) than artificial neural networks (ANNs). This is puzzling given that theoretical results provide exact mapping algorithms from ANNs to SNNs with time-to-first-spike (TTFS) coding. In this paper we analyze in theory and simulation the learning dynamics of TTFS-networks and identify a specific instance of the vanishing-or-exploding gradient problem. While two choices of SNN mappings solve this problem at initialization, only the one with a constant slope of the neuron membrane potential at threshold guarantees the equivalence of the training trajectory between SNNs and ANNs with rectified linear units. We demonstrate that training deep SNN models achieves the exact same performance as that of ANNs, surpassing previous SNNs on image classification datasets such as MNIST/Fashion-MNIST, CIFAR10/CIFAR100 and PLACES365. Our SNN accomplishes high-performance classification with less than 0.3 spikes per neuron, lending itself for an energy-efficient implementation. We show that fine-tuning SNNs with our robust gradient descent algorithm enables their optimization for hardware implementations with low latency and resilience to noise and quantization.