mlsanityData to 7 Oct 2026

Spikformer: When Spiking Neural Network Meets Transformer

arXiv 2209.15425 ↗First posted 29 Sep 2022cs.NE cs.CV cs.LGcited by 263

Not on the map. Why?

Abstract (arXiv)

We consider two biologically plausible structures, the Spiking Neural Network (SNN) and the self-attention mechanism. The former offers an energy-efficient and event-driven paradigm for deep learning, while the latter has the ability to capture feature dependencies, enabling Transformer to achieve good performance. It is intuitively promising to explore the marriage between them. In this paper, we consider leveraging both self-attention capability and biological properties of SNNs, and propose a novel Spiking Self Attention (SSA) as well as a powerful framework, named Spiking Transformer (Spikformer). The SSA mechanism in Spikformer models the sparse visual feature by using spike-form Query, Key, and Value without softmax. Since its computation is sparse and avoids multiplication, SSA is efficient and has low computational energy consumption. It is shown that Spikformer with SSA can outperform the state-of-the-art SNNs-like frameworks in image classification on both neuromorphic and static datasets. Spikformer (66.3M parameters) with comparable size to SEW-ResNet-152 (60.2M,69.26%) can achieve 74.81% top1 accuracy on ImageNet using 4 time steps, which is the state-of-the-art in directly trained SNNs models.

Key facts. Cited by 263 papers in the map (data through 7 Oct 2026). Builds on 44 of its 74 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

44 of 74 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  8. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  11. 2020Pre-Trained Image Processing Transformercited by 891
  12. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
32 more
  1. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 604
  2. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  3. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 399
  4. 2021Early Convolutions Help Transformers See Bettercited by 395
  5. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 340
  6. 2021Deep Residual Learning in Spiking Neural Networkscited by 326
  7. 2020Going Deeper With Directly-Trained Larger Spiking Neural Networkscited by 268
  8. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  9. 2018SLAYER: Spike Layer Error Reassignment in Timecited by 257
  10. 2018Direct Training for Spiking Neural Networks: Faster, Larger, Bettercited by 222
  11. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  12. 2016Training Deep Spiking Neural Networks using Backpropagationcited by 203
  13. 2022Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingcited by 192
  14. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 164
  15. 2020Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagationcited by 142
  16. 2020RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Networkcited by 141
  17. 2019Enabling Spike-based Backpropagation for Training Deep Neural Network Architecturescited by 119
  18. 2021Accurate and efficient time-domain classification with adaptive spiking recurrent neural networkscited by 113
  19. 2021Temporal-wise Attention Spiking Neural Networks for Event Streams Classificationcited by 106
  20. 2022Attention Spiking Neural Networkscited by 103
  21. 2020Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networkscited by 98
  22. 2022Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representationcited by 89
  23. 2020DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networkscited by 70
  24. 2015Spiking Deep Networks with LIF Neuronscited by 63
  25. 2021Relating transformers to models and neural representations of the hippocampal formationcited by 62
  26. 2021HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noisecited by 61
  27. 2020LIAF-Net: Leaky Integrate and Analog Fire Network for Lightweight and Efficient Spatiotemporal Information Processingcited by 56
  28. 2022Neuromorphic Data Augmentation for Training Spiking Neural Networkscited by 52
  29. 2021Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium Statecited by 39
  30. 2022Spiking Graph Convolutional Networkscited by 26
  31. 2020Long Short-Term Memory Spiking Networks and Their Applicationscited by 16
  32. 2021UFO-ViT: High Performance Linear Vision Transformer without Softmaxcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 46; 2024: 64; 2025: 80; 2026: 73.

263 papers.

Sort:

By area:

  1. 2023Spike-driven Transformercited by 141
  2. 2024Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chipscited by 97
  3. 2023SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networkscited by 82
  4. 2024Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Trainingcited by 57
  5. 2023Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacingscited by 48
  6. 2024QKFormer: Hierarchical Spiking Transformer using Q-K Attentioncited by 46
  7. 2023Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependenciescited by 41
  8. 2024SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networkscited by 38
  9. 2023Direct Learning-Based Deep Spiking Neural Networks: A Reviewcited by 31
  10. 2023SpikePoint: An Efficient Point-based Spiking Neural Network for Event Cameras Action Recognitioncited by 30
  11. 2024Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticketcited by 28
  12. 2023RMP-Loss: Regularizing Membrane Potential Distribution for Spiking Neural Networkscited by 27
38 more
  1. 2023Training Full Spike Neural Networks via Auxiliary Accumulation Pathwaycited by 26
  2. 2023Chasing Day and Night: Towards Robust and Efficient All-Day Object Detection Guided by an Event Cameracited by 21
  3. 2024Scalable MatMul-free Language Modelingcited by 21
  4. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  5. 2025Spiking Vision Transformer with Saccadic Attentioncited by 21
  6. 2023SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognitioncited by 20
  7. 2023Learning Long Sequences in Spiking Neural Networkscited by 19
  8. 2024Shrinking Your TimeStep: Towards Low-Latency Neuromorphic Object Recognition with Spiking Neural Networkcited by 19
  9. 2024Spiking-PhysFormer: Camera-Based Remote Photoplethysmography with Parallel Spike-driven Transformercited by 19
  10. 2024SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Modelscited by 19
  11. 2023Spiking Denoising Diffusion Probabilistic Modelscited by 18
  12. 2024Spike2Former: Efficient Spiking Transformer for High-performance Image Segmentationcited by 18
  13. 2024EAS-SNN: End-to-End Adaptive Sampling and Representation for Event-based Detection with Recurrent Spiking Neural Networkscited by 16
  14. 2024Learning A Spiking Neural Network for Efficient Image Derainingcited by 16
  15. 2024SpikeZIP-TF: Conversion is All You Need for Transformer-based SNNcited by 16
  16. 2024DailyDVS-200: A Comprehensive Benchmark Dataset for Event-Based Action Recognitioncited by 16
  17. 2023SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillationcited by 15
  18. 2024Spiking Graph Neural Network on Riemannian Manifoldscited by 15
  19. 2025Spiking Transformer:Introducing Accurate Addition-Only Spiking Self-Attention for Transformercited by 13
  20. 2023Co-learning synaptic delays, weights and adaptation in spiking neural networkscited by 12
  21. 2024Autonomous Driving with Spiking Neural Networkscited by 12
  22. 2025Quantized Spike-driven Transformercited by 12
  23. 2025Rethinking Spiking Neural Networks from an Ensemble Learning Perspectivecited by 12
  24. 2024TIM: An Efficient Temporal Interaction Module for Spiking Transformercited by 11
  25. 2024LMUFormer: Low Complexity Yet Powerful Spiking Model With Legendre Memory Unitscited by 11
  26. 2024Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithmscited by 11
  27. 2025STAA-SNN: Spatial-Temporal Attention Aggregator for Spiking Neural Networkscited by 11
  28. 2025SDTrack: A Baseline for Event-based Tracking via Spiking Neural Networkscited by 11
  29. 2023Highly Efficient 3D Human Pose Tracking from Events with Spiking Spatiotemporal Transformercited by 10
  30. 2023Spiking-Fer: Spiking Neural Network for Facial Expression Recognition With Event Camerascited by 10
  31. 2024LM-HT SNN: Enhancing the Performance of SNN to ANN Counterpart through Learnable Multi-hierarchical Threshold Modelcited by 10
  32. 2025Towards High-performance Spiking Transformers from ANN to SNN Conversioncited by 10
  33. 2025Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networkscited by 10
  34. 2023An Efficient Knowledge Transfer Strategy for Spiking Neural Networks from Static to Event Domaincited by 9
  35. 2023Accurate and Efficient Event-based Semantic Segmentation Using Adaptive Spiking Encoder-Decoder Networkcited by 9
  36. 2023Spikingformer: A Key Foundation Model for Spiking Neural Networkscited by 9
  37. 2024Self-Distillation Learning Based on Temporal-Spatial Consistency for Spiking Neural Networkscited by 9
  38. 2024SPikE-SSM: A Sparse, Precise, and Efficient Spiking State Space Model for Long Sequences Learningcited by 9

50 most cited of 263 All 263 (JSON)