mlsanityData to 14 Oct 2026

Uncovering the Representation of Spiking Neural Networks Trained with Surrogate Gradient

arXiv 2304.13098 ↗First posted 25 Apr 2023cs.LG cs.NEcited by 11

Not on the map. Why?

Abstract (arXiv)

Spiking Neural Networks (SNNs) are recognized as the candidate for the next-generation neural networks due to their bio-plausibility and energy efficiency. Recently, researchers have demonstrated that SNNs are able to achieve nearly state-of-the-art performance in image recognition tasks using surrogate gradient training. However, some essential questions exist pertaining to SNNs that are little studied: Do SNNs trained with surrogate gradient learn different representations from traditional Artificial Neural Networks (ANNs)? Does the time dimension in SNNs provide unique representation power? In this paper, we aim to answer these questions by conducting a representation similarity analysis between SNNs and ANNs using Centered Kernel Alignment (CKA). We start by analyzing the spatial dimension of the networks, including both the width and the depth. Furthermore, our analysis of residual connections shows that SNNs learn a periodic pattern, which rectifies the representations in SNNs to be ANN-like. We additionally investigate the effect of the time dimension on SNN representation, finding that deeper layers encourage more dynamics along the time dimension. We also investigate the impact of input data such as event-stream data and adversarial attacks. Our work uncovers a host of new findings of representations in SNNs. We hope this work will inspire future research to fully comprehend the representation power of SNNs. Code is released at https://github.com/Intelligent-Computing-Lab-Yale/SNNCKA.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 40 of its 64 references that are in the map. First posted on arXiv 25 Apr 2023.

Builds on

40 of 64 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  6. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  8. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  9. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 626
  10. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 406
  11. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 345
  12. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
28 more
  1. 2015Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccadescited by 314
  2. 2020Going Deeper With Directly-Trained Larger Spiking Neural Networkscited by 278
  3. 2018Deep Learning in Spiking Neural Networkscited by 263
  4. 2018Going Deeper in Spiking Neural Networks: VGG and Residual Architecturescited by 245
  5. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 245
  6. 2018Direct Training for Spiking Neural Networks: Faster, Larger, Bettercited by 233
  7. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
  8. 2022Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingcited by 198
  9. 2018Long short-term memory and learning-to-learn in networks of spiking neuronscited by 181
  10. 2020RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Networkcited by 145
  11. 2021Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networkscited by 139
  12. 2021A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibrationcited by 117
  13. 2020Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networkscited by 100
  14. 2019Defensive Quantization: When Efficiency Meets Robustnesscited by 95
  15. 2020Revisiting Batch Normalization for Training Low-latency Deep Spiking Neural Networks from Scratchcited by 80
  16. 2016Supervised learning based on temporal coding in spiking neural networkscited by 76
  17. 2020DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networkscited by 71
  18. 2021HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noisecited by 65
  19. 2018Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representationcited by 62
  20. 2022Neural Architecture Search for Spiking Neural Networkscited by 53
  21. 2022Neuromorphic Data Augmentation for Training Spiking Neural Networkscited by 52
  22. 2020Inherent Adversarial Robustness of Deep Spiking Neural Networks: Effects of Discrete Input Encoding and Non-Linear Activationscited by 48
  23. 2022AutoSNN: Towards Energy-Efficient Spiking Neural Networkscited by 40
  24. 2019A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networkscited by 38
  25. 2016Theory and Tools for the Conversion of Analog to Spiking Convolutional Neural Networkscited by 31
  26. 2020Exploring Adversarial Attack in Spiking Neural Networks with Spike-Compatible Gradientcited by 31
  27. 2021Visual Explanations from Spiking Neural Networks using Interspike Intervalscited by 27
  28. 2022Exploring Lottery Ticket Hypothesis in Spiking Neural Networkscited by 26

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 5; 2025: 4.

11 papers.

  1. 2025Differential Coding for Training-Free ANN-to-SNN Conversioncited by 10
  2. 2024Advancing Training Efficiency of Deep Spiking Neural Networks through Rate-based Backpropagationcited by 8
  3. 2023Adversarially Robust Spiking Neural Networks Through Conversioncited by 7
  4. 2025Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacementcited by 6
  5. 2024Topological Representations of Heterogeneous Learning Dynamics of Recurrent Spiking Neural Networkscited by 3
  6. 2024ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognitioncited by 3
  7. 2023Fast gradient-free activation maximization for neurons in spiking neural networkscited by 2
  8. 2024Benchmarking Spiking Neural Network Learning Methods with Varying Localitycited by 2
  9. 2024Network Representation Learning for Biophysical Neural Network Analysiscited by 0
  10. 2025Random Feature Spiking Neural Networkscited by 0
  11. 2025Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Trainingcited by 0