# Revisiting Batch Normalization for Training Low-latency Deep Spiking Neural Networks from Scratch

arXiv: https://arxiv.org/abs/2010.01729 · HTML: https://mlsanity.com/paper/2010.01729/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2020.

## Built on by (most cited first, 30 of 80)

- [Training Spiking Neural Networks Using Lessons From Deep Learning](https://mlsanity.com/paper/2109.12894/) (2021)
- [Beyond Classification: Directly Training Spiking Neural Networks for Semantic Segmentation](https://mlsanity.com/paper/2110.07742/) (2021)
- [Neural Architecture Search for Spiking Neural Networks](https://mlsanity.com/paper/2201.10355/) (2022)
- [Neuromorphic Data Augmentation for Training Spiking Neural Networks](https://mlsanity.com/paper/2203.06145/) (2022)
- [Fast-SNN: Fast Spiking Neural Network by Converting Quantized ANN](https://mlsanity.com/paper/2305.19868/) (2023)
- [SEENN: Towards Temporal Spiking Early-Exit Neural Networks](https://mlsanity.com/paper/2304.01230/) (2023)
- [Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependencies](https://mlsanity.com/paper/2304.12760/) (2023)
- [AutoSNN: Towards Energy-Efficient Spiking Neural Networks](https://mlsanity.com/paper/2201.12738/) (2022)
- [Bridging the Gap between ANNs and SNNs by Calibrating Offset Spikes](https://mlsanity.com/paper/2302.10685/) (2023)
- [Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorization](https://mlsanity.com/paper/2305.02099/) (2023)
- [Membrane Potential Batch Normalization for Spiking Neural Networks](https://mlsanity.com/paper/2308.08359/) (2023)
- [Direct Learning-Based Deep Spiking Neural Networks: A Review](https://mlsanity.com/paper/2305.19725/) (2023)
- [Visual Explanations from Spiking Neural Networks using Interspike Intervals](https://mlsanity.com/paper/2103.14441/) (2021)
- [Exploring Lottery Ticket Hypothesis in Spiking Neural Networks](https://mlsanity.com/paper/2207.01382/) (2022)
- [Rate Coding or Direct Coding: Which One is Better for Accurate, Robust, and Energy-efficient Spiking Neural Networks?](https://mlsanity.com/paper/2202.03133/) (2022)
- [Exploring Temporal Information Dynamics in Spiking Neural Networks](https://mlsanity.com/paper/2211.14406/) (2022)
- [Federated Learning with Spiking Neural Networks](https://mlsanity.com/paper/2106.06579/) (2021)
- [Backpropagation with Biologically Plausible Spatio-Temporal Adjustment For Training Deep Spiking Neural Networks](https://mlsanity.com/paper/2110.08858/) (2021)
- [One Timestep is All You Need: Training Spiking Neural Networks with Ultra Low Latency](https://mlsanity.com/paper/2110.05929/) (2021)
- [PrivateSNN: Privacy-Preserving Spiking Neural Networks](https://mlsanity.com/paper/2104.03414/) (2021)
- [Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks](https://mlsanity.com/paper/2401.04486/) (2024)
- [Toward Robust Spiking Neural Network Against Adversarial Perturbation](https://mlsanity.com/paper/2205.01625/) (2022)
- [Efficient Federated Learning with Spike Neural Networks for Traffic Sign Recognition](https://mlsanity.com/paper/2205.14315/) (2022)
- [Uncovering the Representation of Spiking Neural Networks Trained with Surrogate Gradient](https://mlsanity.com/paper/2304.13098/) (2023)
- [Input-Aware Dynamic Timestep Spiking Neural Networks for Efficient In-Memory Computing](https://mlsanity.com/paper/2305.17346/) (2023)
- [Keys to Accurate Feature Extraction Using Residual Spiking Neural Networks](https://mlsanity.com/paper/2111.05955/) (2021)
- [Meta-learning Spiking Neural Networks with Surrogate Gradient Descent](https://mlsanity.com/paper/2201.10777/) (2022)
- [MINT: Multiplier-less INTeger Quantization for Energy Efficient Spiking Neural Networks](https://mlsanity.com/paper/2305.09850/) (2023)
- [Enhancing Adversarial Robustness in SNNs with Sparse Gradients](https://mlsanity.com/paper/2405.20355/) (2024)
- [Can Deep Neural Networks be Converted to Ultra Low-Latency Spiking Neural Networks?](https://mlsanity.com/paper/2112.12133/) (2021)

## Abstract (arXiv)

Spiking Neural Networks (SNNs) have recently emerged as an alternative to deep learning owing to sparse, asynchronous and binary event (or spike) driven processing, that can yield huge energy efficiency benefits on neuromorphic hardware. However, training high-accuracy and low-latency SNNs from scratch suffers from non-differentiable nature of a spiking neuron. To address this training issue in SNNs, we revisit batch normalization and propose a temporal Batch Normalization Through Time (BNTT) technique. Most prior SNN works till now have disregarded batch normalization deeming it ineffective for training temporal SNNs. Different from previous works, our proposed BNTT decouples the parameters in a BNTT layer along the time axis to capture the temporal dynamics of spikes. The temporally evolving learnable parameters in BNTT allow a neuron to control its spike rate through different time-steps, enabling low-latency and low-energy training from scratch. We conduct experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet and event-driven DVS-CIFAR10 datasets. BNTT allows us to train deep SNN architectures from scratch, for the first time, on complex datasets with just few 25-30 time-steps. We also propose an early exit algorithm using the distribution of parameters in BNTT to reduce the latency at inference, that further improves the energy-efficiency.