# Toward Efficient Spiking Transformers: Synapse Pruning Meets Synergistic Learning-Based Compensation

arXiv: https://arxiv.org/abs/2508.01992 · HTML: https://mlsanity.com/paper/2508.01992/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 63 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Going Deeper With Directly-Trained Larger Spiking Neural Networks](https://mlsanity.com/paper/2011.05280/) (2020)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting](https://mlsanity.com/paper/2202.11946/) (2022)
- [SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence](https://mlsanity.com/paper/2310.16620/) (2023)
- [Spike-driven Transformer](https://mlsanity.com/paper/2307.01694/) (2023)
- [Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips](https://mlsanity.com/paper/2404.03663/) (2024)
- [Spatio-Temporal Recurrent Networks for Event-Based Optical Flow Estimation](https://mlsanity.com/paper/2109.04871/) (2021)
- [QKFormer: Hierarchical Spiking Transformer using Q-K Attention](https://mlsanity.com/paper/2403.16552/) (2024)
- [Spiking Vision Transformer with Saccadic Attention](https://mlsanity.com/paper/2502.12677/) (2025)
- [Complex Dynamic Neurons Improved Spiking Transformer Network for Efficient Automatic Speech Recognition](https://mlsanity.com/paper/2302.01194/) (2023)
- [Quantized Spike-driven Transformer](https://mlsanity.com/paper/2501.13492/) (2025)
- [Towards High-performance Spiking Transformers from ANN to SNN Conversion](https://mlsanity.com/paper/2502.21193/) (2025)
- [Highly Efficient 3D Human Pose Tracking from Events with Spiking Spatiotemporal Transformer](https://mlsanity.com/paper/2303.09681/) (2023)
- [Spikingformer: A Key Foundation Model for Spiking Neural Networks](https://mlsanity.com/paper/2304.11954/) (2023)
- [Enhancing the Performance of Transformer-based Spiking Neural Networks by SNN-optimized Downsampling with Precise Gradient Backpropagation](https://mlsanity.com/paper/2305.05954/) (2023)
- [A Synapse-Threshold Synergistic Learning Approach for Spiking Neural Networks](https://mlsanity.com/paper/2206.06129/) (2022)
- [Reliable Object Tracking by Multimodal Hybrid Feature Extraction and Transformer-Based Fusion](https://mlsanity.com/paper/2405.17903/) (2024)
- [Spiking Neural Networks for Temporal Processing: Status Quo and Future Prospects](https://mlsanity.com/paper/2502.09449/) (2025)
- [Sharing Leaky-Integrate-and-Fire Neurons for Memory-Efficient Spiking Neural Networks](https://mlsanity.com/paper/2305.18360/) (2023)
- [Spiking World Model with Multi-Compartment Neurons for Model-based Reinforcement Learning](https://mlsanity.com/paper/2503.00713/) (2025)

## Abstract (arXiv)

As a foundational architecture of artificial intelligence models, Transformer has been recently adapted to spiking neural networks with promising performance across various tasks. However, existing spiking Transformer(ST)-based models require a substantial number of parameters and incur high computational costs, thus limiting their deployment in resource-constrained environments. To address these challenges, we propose combining synapse pruning with a synergistic learning-based compensation strategy to derive lightweight ST-based models. Specifically, two types of tailored pruning strategies are introduced to reduce redundancy in the weight matrices of ST blocks: an unstructured mathrmL1P method to induce sparse representations, and a structured DSP method to induce low-rank representations. In addition, we propose an enhanced spiking neuron model, termed the synergistic leaky integrate-and-fire (sLIF) neuron, to effectively compensate for model pruning through synergistic learning between synaptic and intrinsic plasticity mechanisms. Extensive experiments on benchmark datasets demonstrate that the proposed methods significantly reduce model size and computational overhead while maintaining competitive performance. These results validate the effectiveness of the proposed pruning and compensation strategies in constructing efficient and high-performing ST-based models.