# Masked Spiking Transformer

arXiv: https://arxiv.org/abs/2210.01208 · HTML: https://mlsanity.com/paper/2210.01208/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Oct 2022.

## Built on by (most cited first, 30 of 39)

- [Reti-Diff: Illumination Degradation Image Restoration with Retinex-based Latent Diffusion Model](https://mlsanity.com/paper/2311.11638/) (2023)
- [Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training](https://mlsanity.com/paper/2411.16061/) (2024)
- [QKFormer: Hierarchical Spiking Transformer using Q-K Attention](https://mlsanity.com/paper/2403.16552/) (2024)
- [Teaching Tailored to Talent: Adverse Weather Restoration via Prompt Pool and Depth-Anything Constraint](https://mlsanity.com/paper/2409.15739/) (2024)
- [A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models](https://mlsanity.com/paper/2410.07265/) (2024)
- [Chasing Day and Night: Towards Robust and Efficient All-Day Object Detection Guided by an Event Camera](https://mlsanity.com/paper/2309.09297/) (2023)
- [SpikeZIP-TF: Conversion is All You Need for Transformer-based SNN](https://mlsanity.com/paper/2406.03470/) (2024)
- [Learning A Spiking Neural Network for Efficient Image Deraining](https://mlsanity.com/paper/2405.06277/) (2024)
- [Autonomous Driving with Spiking Neural Networks](https://mlsanity.com/paper/2405.19687/) (2024)
- [Quantized Spike-driven Transformer](https://mlsanity.com/paper/2501.13492/) (2025)
- [LM-HT SNN: Enhancing the Performance of SNN to ANN Counterpart through Learnable Multi-hierarchical Threshold Model](https://mlsanity.com/paper/2402.00411/) (2024)
- [Differential Coding for Training-Free ANN-to-SNN Conversion](https://mlsanity.com/paper/2503.00301/) (2025)
- [Optimising Event-Driven Spiking Neural Network with Regularisation and Cutoff](https://mlsanity.com/paper/2301.09522/) (2023)
- [Spiking Neural Network as Adaptive Event Stream Slicer](https://mlsanity.com/paper/2410.02249/) (2024)
- [Toward Large-scale Spiking Neural Networks: A Comprehensive Survey and Future Directions](https://mlsanity.com/paper/2409.02111/) (2024)
- [Adaptive Calibration: A Unified Conversion Framework of Spiking Neural Network](https://mlsanity.com/paper/2412.16219/) (2024)
- [Spiking Meets Attention: Efficient Remote Sensing Image Super-Resolution with Attention Spiking Neural Networks](https://mlsanity.com/paper/2503.04223/) (2025)
- [RN-Net: Reservoir Nodes-Enabled Neuromorphic Vision Sensing Network](https://mlsanity.com/paper/2303.10770/) (2023)
- [Spiking Neural Networks Need High Frequency Information](https://mlsanity.com/paper/2505.18608/) (2025)
- [Brain-inspired Action Generation with Spiking Transformer Diffusion Policy Model](https://mlsanity.com/paper/2411.09953/) (2024)
- [MSVIT: Improving Spiking Vision Transformer Using Multi-scale Attention Fusion](https://mlsanity.com/paper/2505.14719/) (2025)
- [Edge Intelligence with Spiking Neural Networks](https://mlsanity.com/paper/2507.14069/) (2025)
- [Neural Dynamics Self-Attention for Spiking Transformers](https://mlsanity.com/paper/2603.19290/) (2026)
- [Spiking Pyramid Wavelet Transformation for High-efficient and Low-energy Image Restoration](https://mlsanity.com/paper/2606.18644/) (2026)
- [SDformerFlow: Spatiotemporal swin spikeformer for event-based optical flow estimation](https://mlsanity.com/paper/2409.04082/) (2024)
- [Region Masking to Accelerate Video Processing on Neuromorphic Hardware](https://mlsanity.com/paper/2503.16775/) (2025)
- [SpikeDerain: Unveiling Clear Videos from Rainy Sequences Using Color Spike Streams](https://mlsanity.com/paper/2503.20315/) (2025)
- [IML-Spikeformer: Input-aware Multi-Level Spiking Transformer for Speech Processing](https://mlsanity.com/paper/2507.07396/) (2025)
- [Error Amplification Limits ANN-to-SNN Conversion in Continuous Control](https://mlsanity.com/paper/2601.21778/) (2026)
- [Adaptive Calibration: A Unified Conversion Framework of Spiking Neural Networks](https://mlsanity.com/paper/2311.14265/) (2023)

## Abstract (arXiv)

The combination of Spiking Neural Networks (SNNs) and Transformers has attracted significant attention due to their potential for high energy efficiency and high-performance nature. However, existing works on this topic typically rely on direct training, which can lead to suboptimal performance. To address this issue, we propose to leverage the benefits of the ANN-to-SNN conversion method to combine SNNs and Transformers, resulting in significantly improved performance over existing state-of-the-art SNN models. Furthermore, inspired by the quantal synaptic failures observed in the nervous system, which reduces the number of spikes transmitted across synapses, we introduce a novel Masked Spiking Transformer (MST) framework that incorporates a Random Spike Masking (RSM) method to prune redundant spikes and reduce energy consumption without sacrificing performance. Our experimental results demonstrate that the proposed MST model achieves a significant reduction of 26.8% in power consumption when the masking ratio is 75% while maintaining the same level of performance as the unmasked model.