mlsanityData to 14 Oct 2026

Attention Spiking Neural Networks

arXiv 2209.13929 ↗First posted 28 Sep 2022cs.CVcited by 107

Not on the map. Why?

Abstract (arXiv)

Benefiting from the event-driven and sparse spiking characteristics of the brain, spiking neural networks (SNNs) are becoming an energy-efficient alternative to artificial neural networks (ANNs). However, the performance gap between SNNs and ANNs has been a great hindrance to deploying SNNs ubiquitously for a long time. To leverage the full potential of SNNs, we study the effect of attention mechanisms in SNNs. We first present our idea of attention with a plug-and-play kit, termed the Multi-dimensional Attention (MA). Then, a new attention SNN architecture with end-to-end training called "MA-SNN" is proposed, which infers attention weights along the temporal, channel, as well as spatial dimensions separately or simultaneously. Based on the existing neuroscience theories, we exploit the attention weights to optimize membrane potentials, which in turn regulate the spiking response in a data-dependent way. At the cost of negligible additional parameters, MA facilitates vanilla SNNs to achieve sparser spiking activity, better performance, and energy efficiency concurrently. Experiments are conducted in event-based DVS128 Gesture/Gait action recognition and ImageNet-1k image classification. On Gesture/Gait, the spike counts are reduced by 84.9%/81.6%, and the task accuracy and energy efficiency are improved by 5.9%/4.7% and 3.4×/3.2×. On ImageNet-1K, we achieve top-1 accuracy of 75.92% and 77.08% on single/4-step Res-SNN-104, which are state-of-the-art results in SNNs. To our best knowledge, this is for the first time, that the SNN community achieves comparable or even better performance compared with its ANN counterpart in the large-scale dataset. Our work lights up SNN's potential as a general backbone to support various applications for SNNs, with a great balance between effectiveness and efficiency.

Key facts. Cited by 107 papers in the map (data through 14 Oct 2026). Builds on 43 of its 79 references that are in the map. First posted on arXiv 28 Sep 2022.

Builds on

43 of 79 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2017Squeeze-and-Excitation Networkscited by 4,789
  7. 2018CBAM: Convolutional Block Attention Modulecited by 2,477
  8. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  9. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  10. 2021ViViT: A Video Vision Transformercited by 1,620
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  12. 2019Event-based Vision: A Surveycited by 1,016
31 more
  1. 2020A Survey on Visual Transformercited by 654
  2. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 626
  3. 2019ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networkscited by 575
  4. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 434
  5. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 406
  6. 2017Residual Attention Network for Image Classificationcited by 386
  7. 2019High Speed and High Dynamic Range Video with an Event Cameracited by 345
  8. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 345
  9. 2021Deep Residual Learning in Spiking Neural Networkscited by 331
  10. 2020Going Deeper With Directly-Trained Larger Spiking Neural Networkscited by 278
  11. 2018SLAYER: Spike Layer Error Reassignment in Timecited by 266
  12. 2018Going Deeper in Spiking Neural Networks: VGG and Residual Architecturescited by 245
  13. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  14. 2022Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingcited by 198
  15. 2020RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Networkcited by 145
  16. 2021Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networkscited by 139
  17. 2021A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibrationcited by 117
  18. 2021Accurate and efficient time-domain classification with adaptive spiking recurrent neural networkscited by 115
  19. 2021Temporal-wise Attention Spiking Neural Networks for Event Streams Classificationcited by 106
  20. 2018Global Second-order Pooling Convolutional Networkscited by 66
  21. 2021HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noisecited by 65
  22. 2020Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networkscited by 47
  23. 2019Temporal Coding in Spiking Neural Networks with Alpha Synaptic Function: Learning with Backpropagationcited by 45
  24. 2019Comprehensive SNN Compression Using ADMM Optimization and Activity Regularizationcited by 45
  25. 2022Optimized Potential Initialization for Low-latency Spiking Neural Networkscited by 41
  26. 2021Visual Explanations from Spiking Neural Networks using Interspike Intervalscited by 27
  27. 2021One Timestep is All You Need: Training Spiking Neural Networks with Ultra Low Latencycited by 17
  28. 2020A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networkscited by 9
  29. 2022Sparse Compressed Spiking Neural Network Accelerator for Object Detectioncited by 6
  30. 2022Rethinking Pretraining as a Bridge from ANNs to SNNscited by 4
  31. 2021Direct Training via Backpropagation for Ultra-low Latency Spiking Neural Networks with Multi-thresholdcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 29; 2024: 29; 2025: 36; 2026: 11.

107 papers.

Sort:

By area:

  1. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 271
  2. 2023SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligencecited by 169
  3. 2023Spike-driven Transformercited by 146
  4. 2024Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chipscited by 103
  5. 2023SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networkscited by 88
  6. 2023Deep Directly-Trained Spiking Neural Networks for Object Detectioncited by 77
  7. 2024Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Trainingcited by 59
  8. 2023Speck: A Smart event-based Vision Sensor with a low latency 327K Neuron Convolutional Neuronal Network Processing Pipelinecited by 52
  9. 2024QKFormer: Hierarchical Spiking Transformer using Q-K Attentioncited by 49
  10. 2023Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependenciescited by 41
  11. 2024Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticketcited by 30
  12. 2023Chasing Day and Night: Towards Robust and Efficient All-Day Object Detection Guided by an Event Cameracited by 21
38 more
  1. 2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Mapcited by 21
  2. 2025Spiking Vision Transformer with Saccadic Attentioncited by 21
  3. 2023Spiking Denoising Diffusion Probabilistic Modelscited by 19
  4. 2023SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillationcited by 17
  5. 2024Learning A Spiking Neural Network for Efficient Image Derainingcited by 16
  6. 2023Inherent Redundancy in Spiking Neural Networkscited by 13
  7. 2023Co-learning synaptic delays, weights and adaptation in spiking neural networkscited by 13
  8. 2024Autonomous Driving with Spiking Neural Networkscited by 13
  9. 2025Spiking Transformer:Introducing Accurate Addition-Only Spiking Self-Attention for Transformercited by 13
  10. 2024LM-HT SNN: Enhancing the Performance of SNN to ANN Counterpart through Learnable Multi-hierarchical Threshold Modelcited by 11
  11. 2025STAA-SNN: Spatial-Temporal Attention Aggregator for Spiking Neural Networkscited by 11
  12. 2023Highly Efficient 3D Human Pose Tracking from Events with Spiking Spatiotemporal Transformercited by 10
  13. 2023Temporal Knowledge Sharing enable Spiking Neural Network Learning from Past and Futurecited by 10
  14. 2024Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learningcited by 10
  15. 2025Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networkscited by 10
  16. 2024SpGesture: Source-Free Domain-adaptive sEMG-based Gesture Recognition with Jaccard Attentive Spiking Neural Networkcited by 9
  17. 2024Reliable Object Tracking by Multimodal Hybrid Feature Extraction and Transformer-Based Fusioncited by 9
  18. 2025Towards Effective and Sparse Adversarial Attack on Spiking Neural Networks via Breaking Invisible Surrogate Gradientscited by 9
  19. 2023Tensor Decomposition Based Attention Module for Spiking Neural Networkscited by 8
  20. 2024Ternary Spike-based Neuromorphic Signal Processing Systemcited by 8
  21. 2024Spiking Neural Network as Adaptive Event Stream Slicercited by 8
  22. 2025DA-LIF: Dual Adaptive Leaky Integrate-and-Fire Model for Deep Spiking Neural Networkscited by 8
  23. 2024Advancing Spiking Neural Networks towards Multiscale Spatiotemporal Interaction Learningcited by 7
  24. 2024Toward Large-scale Spiking Neural Networks: A Comprehensive Survey and Future Directionscited by 7
  25. 2023MT-SNN: Enhance Spiking Neural Network with Multiple Thresholdscited by 6
  26. 2023AutoST: Training-free Neural Architecture Search for Spiking Transformerscited by 6
  27. 2023SpikingNeRF: Making Bio-inspired Neural Networks See through the Real Worldcited by 6
  28. 2023SpikeMOT: Event-based Multi-Object Tracking with Sparse Motion Featurescited by 6
  29. 2023Fully Spiking Neural Network for Legged Robotscited by 6
  30. 2025Spiking Meets Attention: Efficient Remote Sensing Image Super-Resolution with Attention Spiking Neural Networkscited by 6
  31. 2025TS-LIF: A Temporal Segment Spiking Neuron Network for Time Series Forecastingcited by 6
  32. 2025Event Camera Meets Mobile Embodied Perception: Abstraction, Algorithm, Acceleration, Applicationcited by 6
  33. 2024Sign Gradient Descent-based Neuronal Dynamics: ANN-to-SNN Conversion Beyond ReLU Networkcited by 5
  34. 2024Degree-Conscious Spiking Graph for Cross-Domain Adaptationcited by 5
  35. 2025TS-SNN: Temporal Shift Module for Spiking Neural Networkscited by 5
  36. 2023SpikeCLIP: A Contrastive Language-Image Pretrained Spiking Neural Networkcited by 4
  37. 2025SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and O(T) Complexitycited by 4
  38. 2026TP-Spikformer: Token Pruned Spiking Transformercited by 4

50 most cited of 107 All 107 (JSON)