# Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting

arXiv: https://arxiv.org/abs/2202.11946 · HTML: https://mlsanity.com/paper/2202.11946/

## Key facts

- Cited by 198 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 24 Feb 2022.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [Deep Residual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2102.04159/) (2021)
- [Going Deeper With Directly-Trained Larger Spiking Neural Networks](https://mlsanity.com/paper/2011.05280/) (2020)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [Going Deeper in Spiking Neural Networks: VGG and Residual Architectures](https://mlsanity.com/paper/1802.02627/) (2018)
- [Direct Training for Spiking Neural Networks: Faster, Larger, Better](https://mlsanity.com/paper/1809.05793/) (2018)
- [Training Deep Spiking Neural Networks using Backpropagation](https://mlsanity.com/paper/1608.08782/) (2016)
- [Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection](https://mlsanity.com/paper/1903.06530/) (2019)
- [RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Network](https://mlsanity.com/paper/2003.01811/) (2020)
- [Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation](https://mlsanity.com/paper/2005.01807/) (2020)
- [Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks](https://mlsanity.com/paper/2103.00476/) (2021)
- [A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibration](https://mlsanity.com/paper/2106.06984/) (2021)
- [Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networks](https://mlsanity.com/paper/2002.10085/) (2020)
- [DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networks](https://mlsanity.com/paper/2008.03658/) (2020)
- [Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing](https://mlsanity.com/paper/1603.08270/) (2016)
- [LIAF-Net: Leaky Integrate and Analog Fire Network for Lightweight and Efficient Spatiotemporal Information Processing](https://mlsanity.com/paper/2011.06176/) (2020)
- [Neural Architecture Search for Spiking Neural Networks](https://mlsanity.com/paper/2201.10355/) (2022)
- [On the insufficiency of existing momentum schemes for Stochastic Optimization](https://mlsanity.com/paper/1803.05591/) (2018)
- [Conversion of Artificial Recurrent Neural Networks to Spiking Neural Networks for Low-power Neuromorphic Hardware](https://mlsanity.com/paper/1601.04187/) (2016)
- [Theory and Tools for the Conversion of Analog to Spiking Convolutional Neural Networks](https://mlsanity.com/paper/1612.04052/) (2016)
- [Visual Explanations from Spiking Neural Networks using Interspike Intervals](https://mlsanity.com/paper/2103.14441/) (2021)
- [Convolutional Spiking Neural Networks for Spatio-Temporal Feature Extraction](https://mlsanity.com/paper/2003.12346/) (2020)
- [Backpropagated Neighborhood Aggregation for Accurate Training of Spiking Neural Networks](https://mlsanity.com/paper/2107.06861/) (2021)

## Built on by (most cited first, 30 of 198)

- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [Spike-driven Transformer](https://mlsanity.com/paper/2307.01694/) (2023)
- [Attention Spiking Neural Networks](https://mlsanity.com/paper/2209.13929/) (2022)
- [GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networks](https://mlsanity.com/paper/2210.13768/) (2022)
- [Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training](https://mlsanity.com/paper/2411.16061/) (2024)
- [Neuromorphic Data Augmentation for Training Spiking Neural Networks](https://mlsanity.com/paper/2203.06145/) (2022)
- [Online Training Through Time for Spiking Neural Networks](https://mlsanity.com/paper/2210.04195/) (2022)
- [QKFormer: Hierarchical Spiking Transformer using Q-K Attention](https://mlsanity.com/paper/2403.16552/) (2024)
- [Fast-SNN: Fast Spiking Neural Network by Converting Quantized ANN](https://mlsanity.com/paper/2305.19868/) (2023)
- [SEENN: Towards Temporal Spiking Early-Exit Neural Networks](https://mlsanity.com/paper/2304.01230/) (2023)
- [Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependencies](https://mlsanity.com/paper/2304.12760/) (2023)
- [Reducing ANN-SNN Conversion Error through Residual Membrane Potential](https://mlsanity.com/paper/2302.02091/) (2023)
- [Bridging the Gap between ANNs and SNNs by Calibrating Offset Spikes](https://mlsanity.com/paper/2302.10685/) (2023)
- [Ternary Spike: Learning Ternary Spikes for Spiking Neural Networks](https://mlsanity.com/paper/2312.06372/) (2023)
- [Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks](https://mlsanity.com/paper/2302.14311/) (2023)
- [TCJA-SNN: Temporal-Channel Joint Attention for Spiking Neural Networks](https://mlsanity.com/paper/2206.10177/) (2022)
- [Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorization](https://mlsanity.com/paper/2305.02099/) (2023)
- [Membrane Potential Batch Normalization for Spiking Neural Networks](https://mlsanity.com/paper/2308.08359/) (2023)
- [Direct Learning-Based Deep Spiking Neural Networks: A Review](https://mlsanity.com/paper/2305.19725/) (2023)
- [Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticket](https://mlsanity.com/paper/2401.02020/) (2024)
- [RMP-Loss: Regularizing Membrane Potential Distribution for Spiking Neural Networks](https://mlsanity.com/paper/2308.06787/) (2023)
- [Exploring Lottery Ticket Hypothesis in Spiking Neural Networks](https://mlsanity.com/paper/2207.01382/) (2022)
- [Training Full Spike Neural Networks via Auxiliary Accumulation Pathway](https://mlsanity.com/paper/2301.11929/) (2023)
- [V2CE: Video to Continuous Events Simulator](https://mlsanity.com/paper/2309.08891/) (2023)
- [Exploring Temporal Information Dynamics in Spiking Neural Networks](https://mlsanity.com/paper/2211.14406/) (2022)
- [ESL-SNNs: An Evolutionary Structure Learning Strategy for Spiking Neural Networks](https://mlsanity.com/paper/2306.03693/) (2023)
- [Chasing Day and Night: Towards Robust and Efficient All-Day Object Detection Guided by an Event Camera](https://mlsanity.com/paper/2309.09297/) (2023)
- [Spiking Vision Transformer with Saccadic Attention](https://mlsanity.com/paper/2502.12677/) (2025)
- [STSC-SNN: Spatio-Temporal Synaptic Connection with Temporal Convolution and Attention for Spiking Neural Networks](https://mlsanity.com/paper/2210.05241/) (2022)
- [Spiking Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2306.17046/) (2023)

## Abstract (arXiv)

Recently, brain-inspired spiking neuron networks (SNNs) have attracted widespread research interest because of their event-driven and energy-efficient characteristics. Still, it is difficult to efficiently train deep SNNs due to the non-differentiability of its activation function, which disables the typically used gradient descent approaches for traditional artificial neural networks (ANNs). Although the adoption of surrogate gradient (SG) formally allows for the back-propagation of losses, the discrete spiking mechanism actually differentiates the loss landscape of SNNs from that of ANNs, failing the surrogate gradient methods to achieve comparable accuracy as for ANNs. In this paper, we first analyze why the current direct training approach with surrogate gradient results in SNNs with poor generalizability. Then we introduce the temporal efficient training (TET) approach to compensate for the loss of momentum in the gradient descent with SG so that the training process can converge into flatter minima with better generalizability. Meanwhile, we demonstrate that TET improves the temporal scalability of SNN and induces a temporal inheritable training for acceleration. Our method consistently outperforms the SOTA on all reported mainstream datasets, including CIFAR-10/100 and ImageNet. Remarkably on DVS-CIFAR10, we obtained 83% top-1 accuracy, over 10% improvement compared to existing state of the art. Codes are available at urlhttps://github.com/Gus-Lab/temporalefficienttraining.