# Binary Event-Driven Spiking Transformer

arXiv: https://arxiv.org/abs/2501.05904 · HTML: https://mlsanity.com/paper/2501.05904/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 10 Jan 2025.

## Builds on (most cited first, 16 of 16)

- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm](https://mlsanity.com/paper/1808.00278/) (2018)
- [The Reversible Residual Network: Backpropagation Without Storing Activations](https://mlsanity.com/paper/1707.04585/) (2017)
- [Binary Neural Networks: A Survey](https://mlsanity.com/paper/2004.03333/) (2020)
- [Forward and Backward Information Retention for Accurate Binary Neural Networks](https://mlsanity.com/paper/1909.10788/) (2019)
- [Spike-driven Transformer](https://mlsanity.com/paper/2307.01694/) (2023)
- [Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer](https://mlsanity.com/paper/2210.06707/) (2022)
- [Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips](https://mlsanity.com/paper/2404.03663/) (2024)
- [Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization](https://mlsanity.com/paper/1911.00822/) (2019)
- [Ternary Spike: Learning Ternary Spikes for Spiking Neural Networks](https://mlsanity.com/paper/2312.06372/) (2023)
- [Q-DETR: An Efficient Low-Bit Quantized Detection Transformer](https://mlsanity.com/paper/2304.00253/) (2023)
- [Spikeformer: A Novel Architecture for Training High-Performance Low-Latency Spiking Neural Network](https://mlsanity.com/paper/2211.10686/) (2022)
- [Q-SNNs: Quantized Spiking Neural Networks](https://mlsanity.com/paper/2406.13672/) (2024)
- [MINT: Multiplier-less INTeger Quantization for Energy Efficient Spiking Neural Networks](https://mlsanity.com/paper/2305.09850/) (2023)

## Built on by (most cited first, 3 of 3)

- [TP-Spikformer: Token Pruned Spiking Transformer](https://mlsanity.com/paper/2603.00527/) (2026)
- [BSO: Binary Spiking Online Optimization Algorithm](https://mlsanity.com/paper/2511.12502/) (2025)
- [Neural Dynamics Self-Attention for Spiking Transformers](https://mlsanity.com/paper/2603.19290/) (2026)

## Abstract (arXiv)

Transformer-based Spiking Neural Networks (SNNs) introduce a novel event-driven self-attention paradigm that combines the high performance of Transformers with the energy efficiency of SNNs. However, the larger model size and increased computational demands of the Transformer structure limit their practicality in resource-constrained scenarios. In this paper, we integrate binarization techniques into Transformer-based SNNs and propose the Binary Event-Driven Spiking Transformer, i.e. BESTformer. The proposed BESTformer can significantly reduce storage and computational demands by representing weights and attention maps with a mere 1-bit. However, BESTformer suffers from a severe performance drop from its full-precision counterpart due to the limited representation capability of binarization. To address this issue, we propose a Coupled Information Enhancement (CIE) method, which consists of a reversible framework and information enhancement distillation. By maximizing the mutual information between the binary model and its full-precision counterpart, the CIE method effectively mitigates the performance degradation of the BESTformer. Extensive experiments on static and neuromorphic datasets demonstrate that our method achieves superior performance to other binary SNNs, showcasing its potential as a compact yet high-performance model for resource-limited edge devices. The repository of this paper is available at https://github.com/CaoHLin/BESTFormer.