mlsanityData to 7 Oct 2026

Spike-driven Transformer

arXiv 2307.01694 ↗First posted 4 Jul 2023cs.NE cs.CVcited by 141

Not on the map. Why?

Abstract (arXiv)

Spiking Neural Networks (SNNs) provide an energy-efficient deep learning option due to their unique spike-based event-driven (i.e., spike-driven) paradigm. In this paper, we incorporate the spike-driven paradigm into Transformer by the proposed Spike-driven Transformer with four unique properties: 1) Event-driven, no calculation is triggered when the input of Transformer is zero; 2) Binary spike communication, all matrix multiplications associated with the spike matrix can be transformed into sparse additions; 3) Self-attention with linear complexity at both token and channel dimensions; 4) The operations between spike-form Query, Key, and Value are mask and addition. Together, there are only sparse addition operations in the Spike-driven Transformer. To this end, we design a novel Spike-Driven Self-Attention (SDSA), which exploits only mask and addition operations without any multiplication, and thus having up to 87.2× lower computation energy than vanilla self-attention. Especially in SDSA, the matrix multiplication between Query, Key, and Value is designed as the mask operation. In addition, we rearrange all residual connections in the vanilla Transformer before the activation functions to ensure that all neurons transmit binary spike signals. It is shown that the Spike-driven Transformer can achieve 77.1% top-1 accuracy on ImageNet-1K, which is the state-of-the-art result in the SNN field. The source code is available at https://github.com/BICLab/Spike-Driven-Transformer.

Key facts. Cited by 141 papers in the map (data through 7 Oct 2026). Builds on 63 of its 100 references that are in the map. First posted on arXiv 4 Jul 2023.

Builds on

63 of 100 references.

References · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2017Squeeze-and-Excitation Networkscited by 4,763
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  12. 2020Rethinking Attention with Performerscited by 1,584
38 more
  1. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  2. 2019Event-based Vision: A Surveycited by 1,010
  3. 2021Transformers in Vision: A Surveycited by 1,009
  4. 2019Are Sixteen Heads Really Better than One?cited by 803
  5. 2020Efficient Transformers: A Surveycited by 784
  6. 2020A Survey on Visual Transformercited by 653
  7. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 604
  8. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  9. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 399
  10. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  11. 2021Random Feature Attentioncited by 350
  12. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 340
  13. 2021Deep Residual Learning in Spiking Neural Networkscited by 326
  14. 2020Going Deeper With Directly-Trained Larger Spiking Neural Networkscited by 268
  15. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 263
  16. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  17. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  18. 2018Direct Training for Spiking Neural Networks: Faster, Larger, Bettercited by 222
  19. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  20. 2022Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingcited by 192
  21. 2018Long short-term memory and learning-to-learn in networks of spiking neuronscited by 179
  22. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  23. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  24. 2020Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagationcited by 142
  25. 2021Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networkscited by 134
  26. 2021Accurate and efficient time-domain classification with adaptive spiking recurrent neural networkscited by 113
  27. 2021Temporal-wise Attention Spiking Neural Networks for Event Streams Classificationcited by 106
  28. 2022Attention Spiking Neural Networkscited by 103
  29. 2022Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representationcited by 89
  30. 2023SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networkscited by 82
  31. 2021Advancing Spiking Neural Networks towards Deep Residual Learningcited by 76
  32. 2023SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applicationscited by 66
  33. 2022GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networkscited by 63
  34. 2021HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noisecited by 61
  35. 2022Hydra Attention: Efficient Attention with Many Headscited by 55
  36. 2020Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networkscited by 46
  37. 2019Comprehensive SNN Compression Using ADMM Optimization and Activity Regularizationcited by 38
  38. 2021A Long Short-Term Memory for AI Applications in Spike-based Neuromorphic Hardwarecited by 32

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 39; 2025: 54; 2026: 45.

141 papers.

Sort:

By area:

  1. 2024Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chipscited by 97
  2. 2024Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Trainingcited by 57
  3. 2024QKFormer: Hierarchical Spiking Transformer using Q-K Attentioncited by 46
  4. 2024SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networkscited by 38
  5. 2024SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanismscited by 31
  6. 2024SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spikingcited by 29
  7. 2024Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticketcited by 28
  8. 2024Scalable MatMul-free Language Modelingcited by 21
  9. 2025Spiking Vision Transformer with Saccadic Attentioncited by 21
  10. 2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Mapcited by 20
  11. 2023Learning Long Sequences in Spiking Neural Networkscited by 19
  12. 2024Learning A Spiking Neural Network for Efficient Image Derainingcited by 16
38 more
  1. 2024SpikeZIP-TF: Conversion is All You Need for Transformer-based SNNcited by 16
  2. 2023Inherent Redundancy in Spiking Neural Networkscited by 13
  3. 2025Spiking Transformer:Introducing Accurate Addition-Only Spiking Self-Attention for Transformercited by 13
  4. 2024Autonomous Driving with Spiking Neural Networkscited by 12
  5. 2024CAS-ViT: Convolutional Additive Self-attention Vision Transformers for Efficient Mobile Applicationscited by 12
  6. 2025Quantized Spike-driven Transformercited by 12
  7. 2025Rethinking Spiking Neural Networks from an Ensemble Learning Perspectivecited by 12
  8. 2024LMUFormer: Low Complexity Yet Powerful Spiking Model With Legendre Memory Unitscited by 11
  9. 2024Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithmscited by 11
  10. 2025STAA-SNN: Spatial-Temporal Attention Aggregator for Spiking Neural Networkscited by 11
  11. 2025SDTrack: A Baseline for Event-based Tracking via Spiking Neural Networkscited by 11
  12. 2024LM-HT SNN: Enhancing the Performance of SNN to ANN Counterpart through Learnable Multi-hierarchical Threshold Modelcited by 10
  13. 2025Towards High-performance Spiking Transformers from ANN to SNN Conversioncited by 10
  14. 2025Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networkscited by 10
  15. 2024Libra: Building Decoupled Vision System on Large Language Modelscited by 9
  16. 2024Self-Distillation Learning Based on Temporal-Spatial Consistency for Spiking Neural Networkscited by 9
  17. 2025QSViT: A Methodology for Quantizing Spiking Vision Transformerscited by 8
  18. 2024Advancing Spiking Neural Networks towards Multiscale Spatiotemporal Interaction Learningcited by 7
  19. 2024LoAS: Fully Temporal-Parallel Dataflow for Dual-Sparse Spiking Neural Networkscited by 7
  20. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 7
  21. 2024Toward Large-scale Spiking Neural Networks: A Comprehensive Survey and Future Directionscited by 6
  22. 2024Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Modelcited by 6
  23. 2024MK-SGN: A Spiking Graph Convolutional Network with Multimodal Fusion and Knowledge Distillation for Skeleton-based Action Recognitioncited by 5
  24. 2025SpikingBrain: Spiking Brain-inspired Large Modelscited by 5
  25. 2024Neuronal Competition Groups with Supervised STDP for Spike-Based Classificationcited by 4
  26. 2025Satellite Edge Artificial Intelligence with Large Models: Architectures and Technologiescited by 4
  27. 2025SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and O(T) Complexitycited by 4
  28. 2026SpikeTrack: A Spike-driven Framework for Efficient Visual Trackingcited by 4
  29. 2026TP-Spikformer: Token Pruned Spiking Transformercited by 4
  30. 2023Recent Advances in Scalable Energy-Efficient and Trustworthy Spiking Neural networks: from Algorithms to Technologycited by 3
  31. 2024SpikeGraphormer: A High-Performance Graph Transformer with Spiking Graph Attentioncited by 3
  32. 2024SVFormer: A Direct Training Spiking Transformer for Efficient Video Action Recognitioncited by 3
  33. 2024ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognitioncited by 3
  34. 2025Binary Event-Driven Spiking Transformercited by 3
  35. 2025SpiLiFormer: Enhancing Spiking Transformers with Lateral Inhibitioncited by 3
  36. 2025GazeSCRNN: Event-based Near-eye Gaze Tracking using a Spiking Neural Networkcited by 3
  37. 2025Efficient Spiking Point Mamba for Point Cloud Analysiscited by 3
  38. 2025Spiking Neural Networks Need High Frequency Informationcited by 3

50 most cited of 141 All 141 (JSON)