Neural Dynamics Self-Attention for Spiking Transformers
Not on the map. Why?
Abstract (arXiv)
Integrating Spiking Neural Networks (SNNs) with Transformer architectures offers a promising pathway to balance energy efficiency and performance, particularly for edge vision applications. However, existing Spiking Transformers face two critical challenges: (i) a substantial performance gap compared to their Artificial Neural Networks (ANNs) counterparts and (ii) high memory overhead during inference. Through theoretical analysis, we attribute both limitations to the Spiking Self-Attention (SSA) mechanism: the lack of locality bias and the need to store large attention matrices. Inspired by the localized receptive fields (LRF) and membrane-potential dynamics of biological visual neurons, we propose LRF-Dyn, which uses spiking neurons with localized receptive fields to compute attention while reducing memory requirements. Specifically, we introduce a LRF method into SSA to assign higher weights to neighboring regions, strengthening local modeling and improving performance. Building on this, we approximate the resulting attention computation via charge-fire-reset dynamics, eliminating explicit attention-matrix storage and reducing inference-time memory. Extensive experiments on visual tasks confirm that our method reduces memory overhead while delivering significant performance improvements. These results establish it as a key unit for achieving energy-efficient Spiking Transformers.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 41 of its 62 references that are in the map. First posted on arXiv 9 Mar 2026.
Builds on
41 of 62 references.
References · 41
- 2017Attention Is All You Needcited by 59,932
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2020End-to-End Object Detection with Transformerscited by 7,384
- 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
- 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
- 2020Rethinking Attention with Performerscited by 1,584
- 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
- 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
- 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
- 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
29 more
- 2021MetaFormer Is Actually What You Need for Visioncited by 523
- 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
- 2021Intriguing Properties of Vision Transformerscited by 446
- 2018Efficient Attention: Attention with Linear Complexitiescited by 366
- 2021Deep Residual Learning in Spiking Neural Networkscited by 326
- 2022Spikformer: When Spiking Neural Network Meets Transformercited by 263
- 2023Spike-driven Transformercited by 141
- 2021Beyond Self-attention: External Attention using Two Linear Layers for Visual Taskscited by 109
- 2024Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chipscited by 97
- 2021Advancing Spiking Neural Networks towards Deep Residual Learningcited by 76
- 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
- 2024Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Trainingcited by 57
- 2024QKFormer: Hierarchical Spiking Transformer using Q-K Attentioncited by 46
- 2022Masked Spiking Transformercited by 38
- 2024SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networkscited by 38
- 2022EcoFormer: Energy-Saving Attention with Linear Complexitycited by 21
- 2025Spiking Vision Transformer with Saccadic Attentioncited by 21
- 2024SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalizationcited by 17
- 2025SDTrack: A Baseline for Event-based Tracking via Spiking Neural Networkscited by 11
- 2025Towards High-performance Spiking Transformers from ANN to SNN Conversioncited by 10
- 2024You Only Need Less Attention at Each Stage in Vision Transformerscited by 7
- 2024PMSN: A Parallel Multi-compartment Spiking Neuron for Multi-scale Temporal Processingcited by 5
- 2023DISTA: Denoising Spiking Transformer with intrinsic plasticity and spatiotemporal attentioncited by 4
- 2026TP-Spikformer: Token Pruned Spiking Transformercited by 4
- 2025Binary Event-Driven Spiking Transformercited by 3
- 2025Memory-Free and Parallel Computation for Quantized Spiking Neural Networkscited by 2
- 2025Training-Free ANN-to-SNN Conversion for High-Performance Spiking Transformercited by 2
- 2025Dendritic Resonate-and-Fire Neuron for Effective and Efficient Long Sequence Modelingcited by 2
- 2025Unveiling the Spatial-temporal Effective Receptive Fields of Spiking Neural Networkscited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.