mlsanityData to 7 Oct 2026

Contrastive Attention Mitigates Spectral Bias in Spiking Transformers

arXiv 2610.01403 ↗First posted 1 Oct 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Spiking Transformers merge the energy-efficiency of spiking neural networks (SNNs) with the representational power of self-attention, creating a promising architecture for high-performance, energy-efficient computation. However, a performance gap persists versus its counterparts in artificial neural networks (ANNs). Unlike prior works attributing this to binary activations, we reveal that both spiking neurons and spiking self-attention (SSA) act as low-pass filters through multiscale spectral analysis. This characteristic leads to the dissipation of high-frequency components. To address this issue, we propose the Spiking Contrastive Attention (SCA) paradigm, which draw inspiration from the edge-detection and differential sensing properties of biological visual system. By extracting contrast prototypes via global contrastive aggregation and applying local differential refinement, SCA effectively enhances high-frequency information. Extensive experiments show that SCA is a general module that consistently boosts Spiking Transformers across image classification, semantic segmentation, and event-based tracking. Furthermore, it achieves lower complexity, offering superior efficiency over original SSA. These results establish its potential as a fundamental building block for energy-efficient Spiking Transformers.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 67 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

33 of 67 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  6. 2018On the Spectral Bias of Neural Networkscited by 1,321
  7. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  8. 2020ResNeSt: Split-Attention Networkscited by 445
  9. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 399
  10. 2021Learning Spatio-Temporal Transformer for Visual Trackingcited by 348
  11. 2022Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Frameworkcited by 328
  12. 2021Deep Residual Learning in Spiking Neural Networkscited by 326
21 more
  1. 2022How Do Vision Transformers Work?cited by 305
  2. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 263
  3. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  4. 2023Spike-driven Transformercited by 141
  5. 2021VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flowscited by 128
  6. 2023Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networkscited by 116
  7. 2021A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibrationcited by 111
  8. 2021Dirichlet Energy Constrained Learning for Deep Graph Neural Networkscited by 97
  9. 2024Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chipscited by 97
  10. 2021Object Tracking by Jointly Exploiting Frame and Event Domaincited by 83
  11. 2021Advancing Spiking Neural Networks towards Deep Residual Learningcited by 76
  12. 2023HIPTrack: Visual Tracking with Historical Promptscited by 71
  13. 2024Integer-Valued Training and Spike-Driven Inference Spiking Neural Network for High-performance and Energy-efficient Object Detectioncited by 65
  14. 2024Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Trainingcited by 57
  15. 2024QKFormer: Hierarchical Spiking Transformer using Q-K Attentioncited by 46
  16. 2024SpikeZIP-TF: Conversion is All You Need for Transformer-based SNNcited by 16
  17. 2024Event-Driven Learning for Spiking Neural Networkscited by 15
  18. 2025SDTrack: A Baseline for Event-based Tracking via Spiking Neural Networkscited by 11
  19. 2024LM-HT SNN: Enhancing the Performance of SNN to ANN Counterpart through Learnable Multi-hierarchical Threshold Modelcited by 10
  20. 2025Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentialscited by 4
  21. 2025Spiking Neural Networks Need High Frequency Informationcited by 3

Built on by

Not cited yet.