# Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training

arXiv: https://arxiv.org/abs/2411.16061 · HTML: https://mlsanity.com/paper/2411.16061/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 79 of its 112 references that are in the map.
- First posted on arXiv 25 Nov 2024.

## Builds on (most cited first, 30 of 79)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [Bottleneck Transformers for Visual Recognition](https://mlsanity.com/paper/2101.11605/) (2021)
- [Deep Residual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2102.04159/) (2021)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [Going Deeper With Directly-Trained Larger Spiking Neural Networks](https://mlsanity.com/paper/2011.05280/) (2020)

## Built on by (most cited first, 30 of 59)

- [Spike2Former: Efficient Spiking Transformer for High-performance Image Segmentation](https://mlsanity.com/paper/2412.14587/) (2024)
- [Quantized Spike-driven Transformer](https://mlsanity.com/paper/2501.13492/) (2025)
- [SDTrack: A Baseline for Event-based Tracking via Spiking Neural Networks](https://mlsanity.com/paper/2503.08703/) (2025)
- [Efficient 3D Recognition with Event-driven Spike Sparse Convolution](https://mlsanity.com/paper/2412.07360/) (2024)
- [Spiking Meets Attention: Efficient Remote Sensing Image Super-Resolution with Attention Spiking Neural Networks](https://mlsanity.com/paper/2503.04223/) (2025)
- [Fully Spiking Neural Networks for Unified Frame-Event Object Tracking](https://mlsanity.com/paper/2505.20834/) (2025)
- [SpikingBrain: Spiking Brain-inspired Large Models](https://mlsanity.com/paper/2509.05276/) (2025)
- [SpikeTrack: A Spike-driven Framework for Efficient Visual Tracking](https://mlsanity.com/paper/2602.23963/) (2026)
- [Universal Image Restoration Pre-training via Masked Degradation Classification](https://mlsanity.com/paper/2510.13282/) (2025)
- [TP-Spikformer: Token Pruned Spiking Transformer](https://mlsanity.com/paper/2603.00527/) (2026)
- [SpiLiFormer: Enhancing Spiking Transformers with Lateral Inhibition](https://mlsanity.com/paper/2503.15986/) (2025)
- [Hardware, Algorithms, and Applications of the Neuromorphic Vision Sensor: a Review](https://mlsanity.com/paper/2504.08588/) (2025)
- [Spiking Neural Networks Need High Frequency Information](https://mlsanity.com/paper/2505.18608/) (2025)
- [SpikingMamba: Towards Energy-Efficient Large Language Models via Knowledge Distillation from Mamba](https://mlsanity.com/paper/2510.04595/) (2025)
- [Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment](https://mlsanity.com/paper/2501.15925/) (2025)
- [AR-LIF: Adaptive reset leaky integrate-and-fire neuron for spiking neural networks](https://mlsanity.com/paper/2507.20746/) (2025)
- [Training-Free ANN-to-SNN Conversion for High-Performance Spiking Transformer](https://mlsanity.com/paper/2508.07710/) (2025)
- [Breaking the Modality Wall: Time-step Mixup for Efficient Spiking Knowledge Transfer from Static to Event Domain](https://mlsanity.com/paper/2511.12150/) (2025)
- [Efficient Spike-driven Transformer for High-performance Drone-View Geo-Localization](https://mlsanity.com/paper/2512.19365/) (2025)
- [Neural Dynamics Self-Attention for Spiking Transformers](https://mlsanity.com/paper/2603.19290/) (2026)
- [SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inference](https://mlsanity.com/paper/2604.22575/) (2026)
- [SPACE: SPike-Aware Consistency Enhancement for Test-Time Adaptation in Spiking Neural Networks](https://mlsanity.com/paper/2504.02298/) (2025)
- [SPKLIP: Aligning Spike Video Streams with Natural Language](https://mlsanity.com/paper/2505.12656/) (2025)
- [SVL: Empowering Spiking Neural Networks for Efficient 3D Open-World Understanding](https://mlsanity.com/paper/2505.17674/) (2025)
- [IML-Spikeformer: Input-aware Multi-Level Spiking Transformer for Speech Processing](https://mlsanity.com/paper/2507.07396/) (2025)
- [SNNSIR: A Simple Spiking Neural Network for Stereo Image Restoration](https://mlsanity.com/paper/2508.12271/) (2025)
- [Efficient Training of Spiking Neural Networks by Spike-aware Data Pruning](https://mlsanity.com/paper/2510.04098/) (2025)
- [Sparse by Rule: Probability-Based N:M Pruning for Spiking Neural Networks](https://mlsanity.com/paper/2511.12097/) (2025)
- [Fire on Motion: Optimizing Video Pass-bands for Efficient Spiking Action Recognition](https://mlsanity.com/paper/2601.22675/) (2026)
- [BSViT: A Burst Spiking Vision Transformer for Expressive and Efficient Visual Representation Learning](https://mlsanity.com/paper/2604.23165/) (2026)

## Abstract (arXiv)

The ambition of brain-inspired Spiking Neural Networks (SNNs) is to become a low-power alternative to traditional Artificial Neural Networks (ANNs). This work addresses two major challenges in realizing this vision: the performance gap between SNNs and ANNs, and the high training costs of SNNs. We identify intrinsic flaws in spiking neurons caused by binary firing mechanisms and propose a Spike Firing Approximation (SFA) method using integer training and spike-driven inference. This optimizes the spike firing pattern of spiking neurons, enhancing efficient training, reducing power consumption, improving performance, enabling easier scaling, and better utilizing neuromorphic chips. We also develop an efficient spike-driven Transformer architecture and a spike-masked autoencoder to prevent performance degradation during SNN scaling. On ImageNet-1k, we achieve state-of-the-art top-1 accuracy of 78.5%, 79.8%, 84.0%, and 86.2% with models containing 10M, 19M, 83M, and 173M parameters, respectively. For instance, the 10M model outperforms the best existing SNN by 7.2% on ImageNet, with training time acceleration and inference energy efficiency improved by 4.5× and 3.9×, respectively. We validate the effectiveness and efficiency of the proposed method across various tasks, including object detection, semantic segmentation, and neuromorphic vision tasks. This work enables SNNs to match ANN performance while maintaining the low-power advantage, marking a significant step towards SNNs as a general visual backbone. Code is available at https://github.com/BICLab/Spike-Driven-Transformer-V3.