mlsanityData to 7 Oct 2026

SpikeZIP-TF: Conversion is All You Need for Transformer-based SNN

arXiv 2406.03470 ↗First posted 5 Jun 2024cs.NE cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

Spiking neural network (SNN) has attracted great attention due to its characteristic of high efficiency and accuracy. Currently, the ANN-to-SNN conversion methods can obtain ANN on-par accuracy SNN with ultra-low latency (8 time-steps) in CNN structure on computer vision (CV) tasks. However, as Transformer-based networks have achieved prevailing precision on both CV and natural language processing (NLP), the Transformer-based SNNs are still encounting the lower accuracy w.r.t the ANN counterparts. In this work, we introduce a novel ANN-to-SNN conversion method called SpikeZIP-TF, where ANN and SNN are exactly equivalent, thus incurring no accuracy degradation. SpikeZIP-TF achieves 83.82% accuracy on CV dataset (ImageNet) and 93.79% accuracy on NLP dataset (SST-2), which are higher than SOTA Transformer-based SNNs. The code is available in GitHub: https://github.com/Intelligent-Computing-Research-Group/SpikeZIP_transformer

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 32 of its 54 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

32 of 54 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2016Layer Normalizationcited by 5,579
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  11. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  12. 2016Deep Networks with Stochastic Depthcited by 973
20 more
  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  2. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 604
  3. 2019Learned Step Size Quantizationcited by 604
  4. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  5. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 263
  6. 2018SLAYER: Spike Layer Error Reassignment in Timecited by 257
  7. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  8. 2016Training Deep Spiking Neural Networks using Backpropagationcited by 203
  9. 2023Spike-driven Transformercited by 141
  10. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  11. 2023Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networkscited by 116
  12. 2023SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networkscited by 82
  13. 2021Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networkscited by 74
  14. 2023SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiationcited by 47
  15. 2023Fast-SNN: Fast Spiking Neural Network by Converting Quantized ANNcited by 42
  16. 2022Masked Spiking Transformercited by 38
  17. 2023Bridging the Gap between ANNs and SNNs by Calibrating Offset Spikescited by 38
  18. 2024Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticketcited by 28
  19. 2018Simultaneously Optimizing Weight and Quantizer of Ternary Neural Network using Truncated Gaussian Approximationcited by 20
  20. 2023Auto-Spikformer: Spikformer Architecture Searchcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 7.

16 papers.

Sort:

By area:

  1. 2025Rethinking Spiking Neural Networks from an Ensemble Learning Perspectivecited by 12
  2. 2025Differential Coding for Training-Free ANN-to-SNN Conversioncited by 9
  3. 2025Towards Effective and Sparse Adversarial Attack on Spiking Neural Networks via Breaking Invisible Surrogate Gradientscited by 9
  4. 2024Toward Large-scale Spiking Neural Networks: A Comprehensive Survey and Future Directionscited by 6
  5. 2025Edge Intelligence with Spiking Neural Networkscited by 2
  6. 2025Training-Free ANN-to-SNN Conversion for High-Performance Spiking Transformercited by 2
  7. 2026Advancing Direct Training for Spiking Neural Networks with Circulate-Firing Neurons and Learnable Gradientscited by 1
  8. 2024Obtaining Optimal Spiking Neural Network in Sequence Learning via CRNN-SNN Conversioncited by 0
  9. 2025DTA: Dual Temporal-channel-wise Attention for Spiking Neural Networkscited by 0
  10. 2025CSDformer: A Conversion Method for Fully Spike-Driven Transformercited by 0
  11. 2026NEXUS: Bit-Exact ANN-to-SNN Equivalence via Neuromorphic Gate Circuits with Surrogate-Free Trainingcited by 0
  12. 2026Error Amplification Limits ANN-to-SNN Conversion in Continuous Controlcited by 0
4 more
  1. 2026NeuroTrain: Surveying Local Learning Rules for Spiking Neural Networks with an Open Benchmarking Frameworkcited by 0
  2. 2026ELSA: An ELastic SNN Inference Architecture for Efficient Neuromorphic Computingcited by 0
  3. 2026Spike-driven Vision-Language-Action Modelcited by 0
  4. 2026Contrastive Attention Mitigates Spectral Bias in Spiking Transformerscited by 0