mlsanityData to 14 Oct 2026

Spikingformer: A Key Foundation Model for Spiking Neural Networks

arXiv 2304.11954 ↗First posted 24 Apr 2023cs.NE cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Spiking neural networks (SNNs) offer a promising energy-efficient alternative to artificial neural networks, due to their event-driven spiking computation. However, some foundation SNN backbones (including Spikformer and SEW ResNet) suffer from non-spike computations (integer-float multiplications) caused by the structure of their residual connections. These non-spike computations increase SNNs' power consumption and make them unsuitable for deployment on mainstream neuromorphic hardware. In this paper, we analyze the spike-driven behavior of the residual connection methods in SNNs. We then present Spikingformer, a novel spiking transformer backbone that merges the MS Residual connection with Self-Attention in a biologically plausible way to address the non-spike computation challenge in Spikformer while maintaining global modeling capabilities. We evaluate Spikingformer across 13 datasets spanning large static images, neuromorphic data, and natural language tasks, and demonstrate the effectiveness and universality of Spikingformer, setting a vital benchmark for spiking neural networks. In addition, with the spike-driven features and global modeling capabilities, Spikingformer is expected to become a more efficient general-purpose SNN backbone towards energy-efficient artificial intelligence. Code: https://github.com/TheBrainLab/Spikingformer

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 23 of its 41 references that are in the map. First posted on arXiv 24 Apr 2023.

Builds on

23 of 41 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  7. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  8. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  10. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 626
  11. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 522
  12. 2021Early Convolutions Help Transformers See Bettercited by 396
11 more
  1. 2020Going Deeper With Directly-Trained Larger Spiking Neural Networkscited by 278
  2. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 271
  3. 2018SLAYER: Spike Layer Error Reassignment in Timecited by 266
  4. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 264
  5. 2016Training Deep Spiking Neural Networks using Backpropagationcited by 205
  6. 2019ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blockscited by 178
  7. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 165
  8. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  9. 2019Enabling Spike-based Backpropagation for Training Deep Neural Network Architecturescited by 123
  10. 2021HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noisecited by 65
  11. 2023Training Full Spike Neural Networks via Auxiliary Accumulation Pathwaycited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 8.

10 papers.

  1. 2026Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attentioncited by 1
  2. 2025Toward Efficient Spiking Transformers: Synapse Pruning Meets Synergistic Learning-Based Compensationcited by 0
  3. 2025Attention via Synaptic Plasticity is All You Need: A Biologically Inspired Spiking Neuromorphic Transformercited by 0
  4. 2026Ge2mS-T: Multi-Dimensional Grouping for Ultra-High Energy Efficiency in Spiking Transformercited by 0
  5. 2026SAFformer:Improving Spiking Transformer via Active Predictive Filteringcited by 0
  6. 2026AQ4SViT: An Automated Quantization Framework with Search Gating Policy for Compressing Spiking Vision Transformerscited by 0
  7. 2026Rethinking Attention Locality in Spiking Transformerscited by 0
  8. 2026Event-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformerscited by 0
  9. 2026Parameter Efficient Continual Learning for Sparse Event-Based Transformerscited by 0
  10. 2026Rethinking the Tradeoff Between Temporal Encoding and Nonlinear Computation in Spiking Language Modelscited by 0