# Exploring Temporal Information Dynamics in Spiking Neural Networks

arXiv: https://arxiv.org/abs/2211.14406 · HTML: https://mlsanity.com/paper/2211.14406/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 60 references that are in the map.
- First posted on arXiv 26 Nov 2022.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Deep Residual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2102.04159/) (2021)
- [Going Deeper With Directly-Trained Larger Spiking Neural Networks](https://mlsanity.com/paper/2011.05280/) (2020)
- [Going Deeper in Spiking Neural Networks: VGG and Residual Architectures](https://mlsanity.com/paper/1802.02627/) (2018)
- [Direct Training for Spiking Neural Networks: Faster, Larger, Better](https://mlsanity.com/paper/1809.05793/) (2018)
- [Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting](https://mlsanity.com/paper/2202.11946/) (2022)
- [Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures](https://mlsanity.com/paper/1903.06379/) (2019)
- [A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibration](https://mlsanity.com/paper/2106.06984/) (2021)
- [Fisher-Rao Metric, Geometry, and Complexity of Neural Networks](https://mlsanity.com/paper/1711.01530/) (2017)
- [Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networks](https://mlsanity.com/paper/2002.10085/) (2020)
- [Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach](https://mlsanity.com/paper/1806.01316/) (2018)
- [Revisiting Batch Normalization for Training Low-latency Deep Spiking Neural Networks from Scratch](https://mlsanity.com/paper/2010.01729/) (2020)
- [Supervised learning based on temporal coding in spiking neural networks](https://mlsanity.com/paper/1606.08165/) (2016)
- [Neural Architecture Search for Spiking Neural Networks](https://mlsanity.com/paper/2201.10355/) (2022)
- [Inherent Adversarial Robustness of Deep Spiking Neural Networks: Effects of Discrete Input Encoding and Non-Linear Activations](https://mlsanity.com/paper/2003.10399/) (2020)
- [Temporal Coding in Spiking Neural Networks with Alpha Synaptic Function: Learning with Backpropagation](https://mlsanity.com/paper/1907.13223/) (2019)
- [Spiking Neural Networks and Online Learning: An Overview and Perspectives](https://mlsanity.com/paper/1908.08019/) (2019)
- [Exploring Adversarial Attack in Spiking Neural Networks with Spike-Compatible Gradient](https://mlsanity.com/paper/2001.01587/) (2020)

## Built on by (most cited first, 24 of 24)

- [Shrinking Your TimeStep: Towards Low-Latency Neuromorphic Object Recognition with Spiking Neural Network](https://mlsanity.com/paper/2401.01912/) (2024)
- [EAS-SNN: End-to-End Adaptive Sampling and Representation for Event-based Detection with Recurrent Spiking Neural Networks](https://mlsanity.com/paper/2403.12574/) (2024)
- [TIM: An Efficient Temporal Interaction Module for Spiking Transformer](https://mlsanity.com/paper/2401.11687/) (2024)
- [Efficient Parallel Training Methods for Spiking Neural Networks with Constant Time Complexity](https://mlsanity.com/paper/2506.12087/) (2025)
- [Best of Both Worlds: Hybrid SNN-ANN Architecture for Event-based Optical Flow Estimation](https://mlsanity.com/paper/2306.02960/) (2023)
- [Multi-scale Evolutionary Neural Architecture Search for Deep Spiking Neural Networks](https://mlsanity.com/paper/2304.10749/) (2023)
- [ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition](https://mlsanity.com/paper/2409.01564/) (2024)
- [Deep Unsupervised Learning Using Spike-Timing-Dependent Plasticity](https://mlsanity.com/paper/2307.04054/) (2023)
- [Brain-Inspired Efficient Pruning: Exploiting Criticality in Spiking Neural Networks](https://mlsanity.com/paper/2311.16141/) (2023)
- [Benchmarking Spiking Neural Network Learning Methods with Varying Locality](https://mlsanity.com/paper/2402.01782/) (2024)
- [SFedCA: Credit Assignment-Based Active Client Selection Strategy for Spiking Federated Learning](https://mlsanity.com/paper/2406.12200/) (2024)
- [Towards Efficient and Accurate Spiking Neural Networks via Adaptive Bit Allocation](https://mlsanity.com/paper/2506.23717/) (2025)
- [SPARTA: Advancing Sparse Attention in Spiking Neural Networks via Spike-Timing-Based Prioritization](https://mlsanity.com/paper/2508.01646/) (2025)
- [Spatio-Temporal Pruning for Compressed Spiking Large Language Models](https://mlsanity.com/paper/2508.20122/) (2025)
- [Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers](https://mlsanity.com/paper/2510.07924/) (2025)
- [Spiking Neural Networks: The Future of Brain-Inspired Computing](https://mlsanity.com/paper/2510.27379/) (2025)
- [Brain-inspired Evolutionary Architectures for Spiking Neural Networks](https://mlsanity.com/paper/2309.05263/) (2023)
- [Dynamic Graph Condensation](https://mlsanity.com/paper/2506.13099/) (2025)
- [PTS-SNN: A Prompt-Tuned Temporal Shift Spiking Neural Networks for Efficient Speech Emotion Recognition](https://mlsanity.com/paper/2602.08240/) (2026)
- [Temporal Regularization Training: Unleashing the Potential of Spiking Neural Networks](https://mlsanity.com/paper/2506.19256/) (2025)
- [Adaptive Time-step Training for Enhancing Spike-Based Neural Radiance Fields](https://mlsanity.com/paper/2507.23033/) (2025)
- [TDSNNs: Competitive Topographic Deep Spiking Neural Networks for Visual Cortex Modeling](https://mlsanity.com/paper/2508.04270/) (2025)
- [SDSNN: A Single-Timestep Spiking Neural Network with Self-Dropping Neuron and Bayesian Optimization](https://mlsanity.com/paper/2508.10913/) (2025)
- [Controllable Stochastic Quantization Encoding for Adversarially Robust Spiking Neural Networks](https://mlsanity.com/paper/2610.01558/) (2026)

## Abstract (arXiv)

Most existing Spiking Neural Network (SNN) works state that SNNs may utilize temporal information dynamics of spikes. However, an explicit analysis of temporal information dynamics is still missing. In this paper, we ask several important questions for providing a fundamental understanding of SNNs: What are temporal information dynamics inside SNNs? How can we measure the temporal information dynamics? How do the temporal information dynamics affect the overall learning performance? To answer these questions, we estimate the Fisher Information of the weights to measure the distribution of temporal information during training in an empirical manner. Surprisingly, as training goes on, Fisher information starts to concentrate in the early timesteps. After training, we observe that information becomes highly concentrated in earlier few timesteps, a phenomenon we refer to as temporal information concentration. We observe that the temporal information concentration phenomenon is a common learning feature of SNNs by conducting extensive experiments on various configurations such as architecture, dataset, optimization strategy, time constant, and timesteps. Furthermore, to reveal how temporal information concentration affects the performance of SNNs, we design a loss function to change the trend of temporal information. We find that temporal information concentration is crucial to building a robust SNN but has little effect on classification accuracy. Finally, we propose an efficient iterative pruning method based on our observation on temporal information concentration. Code is available at https://github.com/Intelligent-Computing-Lab-Yale/Exploring-Temporal-Information-Dynamics-in-Spiking-Neural-Networks.