mlsanityData to 14 Oct 2026

Enhancing the Performance of Transformer-based Spiking Neural Networks by SNN-optimized Downsampling with Precise Gradient Backpropagation

arXiv 2305.05954 ↗First posted 10 May 2023cs.NE cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Deep spiking neural networks (SNNs) have drawn much attention in recent years because of their low power consumption, biological rationality and event-driven property. However, state-of-the-art deep SNNs (including Spikformer and Spikingformer) suffer from a critical challenge related to the imprecise gradient backpropagation. This problem arises from the improper design of downsampling modules in these networks, and greatly hampering the overall model performance. In this paper, we propose ConvBN-MaxPooling-LIF (CML), an SNN-optimized downsampling with precise gradient backpropagation. We prove that CML can effectively overcome the imprecision of gradient backpropagation from a theoretical perspective. In addition, we evaluate CML on ImageNet, CIFAR10, CIFAR100, CIFAR10-DVS, DVS128-Gesture datasets, and show state-of-the-art performance on all these datasets with significantly enhanced performances compared with Spikingformer. For instance, our model achieves 77.64 % on ImageNet, 96.04 % on CIFAR10, 81.4% on CIFAR10-DVS, with + 1.79% on ImageNet, +1.16% on CIFAR100 compared with Spikingformer.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 33 of its 47 references that are in the map. First posted on arXiv 10 May 2023.

Builds on

33 of 47 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2014Going Deeper with Convolutionscited by 4,241
  7. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  8. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  9. 2018YOLOv3: An Incremental Improvementcited by 2,659
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  12. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 626
21 more
  1. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 406
  2. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 345
  3. 2021Deep Residual Learning in Spiking Neural Networkscited by 331
  4. 2020Going Deeper With Directly-Trained Larger Spiking Neural Networkscited by 278
  5. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 271
  6. 2018Direct Training for Spiking Neural Networks: Faster, Larger, Bettercited by 233
  7. 2022Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingcited by 198
  8. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 165
  9. 2020Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagationcited by 144
  10. 2023Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networkscited by 122
  11. 2021A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibrationcited by 117
  12. 2021Temporal-wise Attention Spiking Neural Networks for Event Streams Classificationcited by 106
  13. 2020Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networkscited by 100
  14. 2022Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representationcited by 89
  15. 2020DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networkscited by 71
  16. 2020LIAF-Net: Leaky Integrate and Analog Fire Network for Lightweight and Efficient Spatiotemporal Information Processingcited by 56
  17. 2021Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium Statecited by 40
  18. 2023Bridging the Gap between ANNs and SNNs by Calibrating Offset Spikescited by 40
  19. 2022TCJA-SNN: Temporal-Channel Joint Attention for Spiking Neural Networkscited by 35
  20. 2022Spikeformer: A Novel Architecture for Training High-Performance Low-Latency Spiking Neural Networkcited by 19
  21. 2023KLIF: An optimized spiking neuron unit for tuning surrogate gradient slope and membrane potentialcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 4; 2026: 1.

10 papers.

  1. 2024QKFormer: Hierarchical Spiking Transformer using Q-K Attentioncited by 49
  2. 2024Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticketcited by 30
  3. 2024BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge Distillationcited by 11
  4. 2025Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networkscited by 10
  5. 2024Toward Large-scale Spiking Neural Networks: A Comprehensive Survey and Future Directionscited by 7
  6. 2023SpikeCLIP: A Contrastive Language-Image Pretrained Spiking Neural Networkcited by 4
  7. 2025Decision SpikeFormer: Spike-Driven Transformer for Decision Makingcited by 1
  8. 2026Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attentioncited by 1
  9. 2025Toward Efficient Spiking Transformers: Synapse Pruning Meets Synergistic Learning-Based Compensationcited by 0
  10. 2025CogniSNN: Enabling Neuron-Expandability, Pathway-Reusability, and Dynamic-Configurability with Random Graph Architectures in Spiking Neural Networkscited by 0