# SpikeZIP-TF: Conversion is All You Need for Transformer-based SNN

arXiv: https://arxiv.org/abs/2406.03470 · HTML: https://mlsanity.com/paper/2406.03470/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 54 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [Training Deep Spiking Neural Networks using Backpropagation](https://mlsanity.com/paper/1608.08782/) (2016)
- [Spike-driven Transformer](https://mlsanity.com/paper/2307.01694/) (2023)
- [Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks](https://mlsanity.com/paper/2303.04347/) (2023)
- [Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective](https://mlsanity.com/paper/2102.00650/) (2021)
- [SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks](https://mlsanity.com/paper/2302.13939/) (2023)
- [Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networks](https://mlsanity.com/paper/2105.11654/) (2021)
- [SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation](https://mlsanity.com/paper/2308.10873/) (2023)
- [Fast-SNN: Fast Spiking Neural Network by Converting Quantized ANN](https://mlsanity.com/paper/2305.19868/) (2023)
- [Bridging the Gap between ANNs and SNNs by Calibrating Offset Spikes](https://mlsanity.com/paper/2302.10685/) (2023)
- [Masked Spiking Transformer](https://mlsanity.com/paper/2210.01208/) (2022)
- [Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticket](https://mlsanity.com/paper/2401.02020/) (2024)

## Built on by (most cited first, 18 of 18)

- [Rethinking Spiking Neural Networks from an Ensemble Learning Perspective](https://mlsanity.com/paper/2502.14218/) (2025)
- [Differential Coding for Training-Free ANN-to-SNN Conversion](https://mlsanity.com/paper/2503.00301/) (2025)
- [Towards Effective and Sparse Adversarial Attack on Spiking Neural Networks via Breaking Invisible Surrogate Gradients](https://mlsanity.com/paper/2503.03272/) (2025)
- [Toward Large-scale Spiking Neural Networks: A Comprehensive Survey and Future Directions](https://mlsanity.com/paper/2409.02111/) (2024)
- [Edge Intelligence with Spiking Neural Networks](https://mlsanity.com/paper/2507.14069/) (2025)
- [Training-Free ANN-to-SNN Conversion for High-Performance Spiking Transformer](https://mlsanity.com/paper/2508.07710/) (2025)
- [Error Amplification Limits ANN-to-SNN Conversion in Continuous Control](https://mlsanity.com/paper/2601.21778/) (2026)
- [Advancing Direct Training for Spiking Neural Networks with Circulate-Firing Neurons and Learnable Gradients](https://mlsanity.com/paper/2605.27412/) (2026)
- [Obtaining Optimal Spiking Neural Network in Sequence Learning via CRNN-SNN Conversion](https://mlsanity.com/paper/2408.09403/) (2024)
- [Noise Adaptor: Enhancing Low-Latency Spiking Neural Networks through Noise-Injected Low-Bit ANN Conversion](https://mlsanity.com/paper/2411.17431/) (2024)
- [DTA: Dual Temporal-channel-wise Attention for Spiking Neural Networks](https://mlsanity.com/paper/2503.10052/) (2025)
- [Quantization Meets Spikes: Nearly Lossless Conversion at the First Timestep via Polarity Multi-Spike Mapping](https://mlsanity.com/paper/2508.14520/) (2025)
- [CSDformer: A Conversion Method for Fully Spike-Driven Transformer](https://mlsanity.com/paper/2509.17461/) (2025)
- [NEXUS: Bit-Exact ANN-to-SNN Equivalence via Neuromorphic Gate Circuits with Surrogate-Free Training](https://mlsanity.com/paper/2601.21279/) (2026)
- [NeuroTrain: Surveying Local Learning Rules for Spiking Neural Networks with an Open Benchmarking Framework](https://mlsanity.com/paper/2605.15058/) (2026)
- [ELSA: An ELastic SNN Inference Architecture for Efficient Neuromorphic Computing](https://mlsanity.com/paper/2605.20802/) (2026)
- [Spike-driven Vision-Language-Action Model](https://mlsanity.com/paper/2609.39514/) (2026)
- [Contrastive Attention Mitigates Spectral Bias in Spiking Transformers](https://mlsanity.com/paper/2610.01403/) (2026)

## Abstract (arXiv)

Spiking neural network (SNN) has attracted great attention due to its characteristic of high efficiency and accuracy. Currently, the ANN-to-SNN conversion methods can obtain ANN on-par accuracy SNN with ultra-low latency (8 time-steps) in CNN structure on computer vision (CV) tasks. However, as Transformer-based networks have achieved prevailing precision on both CV and natural language processing (NLP), the Transformer-based SNNs are still encounting the lower accuracy w.r.t the ANN counterparts. In this work, we introduce a novel ANN-to-SNN conversion method called SpikeZIP-TF, where ANN and SNN are exactly equivalent, thus incurring no accuracy degradation. SpikeZIP-TF achieves 83.82% accuracy on CV dataset (ImageNet) and 93.79% accuracy on NLP dataset (SST-2), which are higher than SOTA Transformer-based SNNs. The code is available in GitHub: https://github.com/Intelligent-Computing-Research-Group/SpikeZIP_transformer