mlsanityData to 7 Oct 2026

Fourier or Wavelet bases as counterpart self-attention in spikformer for efficient visual classification

arXiv 2403.18228 ↗First posted 27 Mar 2024cs.CV cs.LG cs.NEcited by 0

Not on the map. Why?

Abstract (arXiv)

Energy-efficient spikformer has been proposed by integrating the biologically plausible spiking neural network (SNN) and artificial Transformer, whereby the Spiking Self-Attention (SSA) is used to achieve both higher accuracy and lower computational cost. However, it seems that self-attention is not always necessary, especially in sparse spike-form calculation manners. In this paper, we innovatively replace vanilla SSA (using dynamic bases calculating from Query and Key) with spike-form Fourier Transform, Wavelet Transform, and their combinations (using fixed triangular or wavelets bases), based on a key hypothesis that both of them use a set of basis functions for information transformation. Hence, the Fourier-or-Wavelet-based spikformer (FWformer) is proposed and verified in visual classification tasks, including both static image and event-based video datasets. The FWformer can achieve comparable or even higher accuracies (0.4%-1.5%), higher running speed (9%-51% for training and 19%-70% for inference), reduced theoretical energy consumption (20%-25%), and reduced GPU memory usage (4%-26%), compared to the standard spikformer. Our result indicates the continuous refinement of new Transformers, that are inspired either by biological discovery (spike-form), or information theory (Fourier or Wavelet Transform), is promising.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 54 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

36 of 54 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  6. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  7. 2020Pre-Trained Image Processing Transformercited by 891
  8. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  9. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 399
  10. 2021Early Convolutions Help Transformers See Bettercited by 395
  11. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  12. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 340
24 more
  1. 2021Deep Residual Learning in Spiking Neural Networkscited by 326
  2. 2020Going Deeper With Directly-Trained Larger Spiking Neural Networkscited by 268
  3. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 263
  4. 2018SLAYER: Spike Layer Error Reassignment in Timecited by 257
  5. 2018Direct Training for Spiking Neural Networks: Faster, Larger, Bettercited by 222
  6. 2022Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingcited by 192
  7. 2023SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligencecited by 163
  8. 2020Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagationcited by 142
  9. 2020RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Networkcited by 141
  10. 2023Spike-driven Transformercited by 141
  11. 2019Enabling Spike-based Backpropagation for Training Deep Neural Network Architecturescited by 119
  12. 2021Temporal-wise Attention Spiking Neural Networks for Event Streams Classificationcited by 106
  13. 2020Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networkscited by 98
  14. 2022Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representationcited by 89
  15. 2020DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networkscited by 70
  16. 2020LIAF-Net: Leaky Integrate and Analog Fire Network for Lightweight and Efficient Spatiotemporal Information Processingcited by 56
  17. 2022Neuromorphic Data Augmentation for Training Spiking Neural Networkscited by 52
  18. 2020Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Controlcited by 44
  19. 2020Hard-Coded Gaussian Attention for Neural Machine Translationcited by 42
  20. 2022Multi-stage image denoising with the wavelet transformcited by 37
  21. 2023Training Full Spike Neural Networks via Auxiliary Accumulation Pathwaycited by 26
  22. 2021Exploiting Spiking Dynamics with Spatial-temporal Feature Normalization in Graph Learningcited by 21
  23. 2023Complex Dynamic Neurons Improved Spiking Transformer Network for Efficient Automatic Speech Recognitioncited by 19
  24. 2020Long Short-Term Memory Spiking Networks and Their Applicationscited by 16

Built on by

Not cited yet.