mlsanityData to 14 Oct 2026

Temporal Regularization Training: Unleashing the Potential of Spiking Neural Networks

arXiv 2506.19256 ↗First posted 24 Jun 2025cs.NE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Spiking Neural Networks (SNNs) have received widespread attention due to their event-driven and low-power characteristics, making them particularly effective for processing neuromorphic data. Recent studies have shown that directly trained SNNs suffer from severe temporal gradient vanishing and overfitting issues, which fundamentally constrain their performance and generalizability. This paper unveils a temporal regularization training (TRT) memthod, designed to unleash the generalization and performance potential of SNNs through a time-decaying regularization mechanism that prioritizes early timesteps with stronger constraints. We perform theoretical analysis to reveal TRT's ability on mitigating the temporal gradient vanishment. To validate the effectiveness of TRT, we conduct experiments on both static image datasets and dynamic neuromorphic datasets, perform analysis of their results, demonstrating that TRT can effectively mitigate overfitting and help SNNs converge into flatter local minima with better generalizability. Furthermore, we establish a theoretical interpretation of TRT's temporal regularization mechanism by analyzing the temporal information dynamics inside SNNs. We track the Fisher information of SNNs during training process, showing that Fisher information progressively concentrates in early timesteps. The time-decaying regularization mechanism implemented in TRT effectively guides the network to learn robust features in early timesteps with rich information, thereby leading to significant improvements in model generalization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 51 references that are in the map. First posted on arXiv 24 Jun 2025.

Builds on

30 of 51 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  5. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 626
  6. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 406
  7. 2018How Does Batch Normalization Help Optimization?cited by 357
  8. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 345
  9. 2021Deep Residual Learning in Spiking Neural Networkscited by 331
  10. 2015Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccadescited by 314
  11. 2020Going Deeper With Directly-Trained Larger Spiking Neural Networkscited by 278
  12. 2018Direct Training for Spiking Neural Networks: Faster, Larger, Bettercited by 233
18 more
  1. 2016Training Deep Spiking Neural Networks using Backpropagationcited by 205
  2. 2022Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingcited by 198
  3. 2020RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Networkcited by 145
  4. 2021Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networkscited by 139
  5. 2021A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibrationcited by 117
  6. 2016Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computingcited by 70
  7. 2020Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networkscited by 43
  8. 2021SpikeDyn: A Framework for Energy-Efficient Spiking Neural Networks with Continual and Unsupervised Learning Capabilities in Dynamic Environmentscited by 29
  9. 2023Gated Attention Coding for Training High-performance and Efficient Spiking Neural Networkscited by 26
  10. 2021ES-ImageNet: A Million Event-Stream Classification Dataset for Spiking Neural Networkscited by 24
  11. 2022Exploring Temporal Information Dynamics in Spiking Neural Networkscited by 24
  12. 2017ASP: Learning to Forget with Adaptive Synaptic Plasticity in Spiking Neural Networkscited by 16
  13. 2023Critical Learning Periods Emerge Even in Deep Linear Networkscited by 13
  14. 2020An electronic neuromorphic system for real-time detection of High Frequency Oscillations (HFOs) in intracranial EEGcited by 11
  15. 2025Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networkscited by 10
  16. 2024Optimal ANN-SNN Conversion with Group Neuronscited by 2
  17. 2024Temporal Reversal Regularization for Spiking Neural Networks: Hybrid Spatio-Temporal Invariance for Generalizationcited by 2
  18. 2020On robot compliance. A cerebellar control approachcited by 1

Built on by

Not cited yet.