# Tensor Decomposition Based Attention Module for Spiking Neural Networks

arXiv: https://arxiv.org/abs/2310.14576 · HTML: https://mlsanity.com/paper/2310.14576/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 69 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Deep Residual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2102.04159/) (2021)
- [Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades](https://mlsanity.com/paper/1507.07629/) (2015)
- [Going Deeper With Directly-Trained Larger Spiking Neural Networks](https://mlsanity.com/paper/2011.05280/) (2020)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [Tensorizing Neural Networks](https://mlsanity.com/paper/1509.06569/) (2015)
- [Direct Training for Spiking Neural Networks: Faster, Larger, Better](https://mlsanity.com/paper/1809.05793/) (2018)
- [Tensor Decomposition for Signal Processing and Machine Learning](https://mlsanity.com/paper/1607.01668/) (2016)
- [Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting](https://mlsanity.com/paper/2202.11946/) (2022)
- [Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation](https://mlsanity.com/paper/2005.01807/) (2020)
- [Attention Spiking Neural Networks](https://mlsanity.com/paper/2209.13929/) (2022)
- [Temporal-wise Attention Spiking Neural Networks for Event Streams Classification](https://mlsanity.com/paper/2107.11711/) (2021)
- [Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation](https://mlsanity.com/paper/2205.00459/) (2022)
- [Deep Multi-task Representation Learning: A Tensor Factorisation Approach](https://mlsanity.com/paper/1605.06391/) (2016)
- [LIAF-Net: Leaky Integrate and Analog Fire Network for Lightweight and Efficient Spatiotemporal Information Processing](https://mlsanity.com/paper/2011.06176/) (2020)
- [Neural Architecture Search for Spiking Neural Networks](https://mlsanity.com/paper/2201.10355/) (2022)
- [Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks](https://mlsanity.com/paper/2007.01204/) (2020)
- [Tensor Regression Networks](https://mlsanity.com/paper/1707.08308/) (2017)
- [Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion Estimation](https://mlsanity.com/paper/1906.06196/) (2019)
- [TCJA-SNN: Temporal-Channel Joint Attention for Spiking Neural Networks](https://mlsanity.com/paper/2206.10177/) (2022)
- [Spike-Train Level Backpropagation for Training Deep Recurrent Spiking Neural Networks](https://mlsanity.com/paper/1908.06378/) (2019)
- [Tensor Low-Rank Reconstruction for Semantic Segmentation](https://mlsanity.com/paper/2008.00490/) (2020)
- [Gated Attention Coding for Training High-performance and Efficient Spiking Neural Networks](https://mlsanity.com/paper/2308.06582/) (2023)
- [Large Separable Kernel Attention: Rethinking the Large Kernel Attention Design in CNN](https://mlsanity.com/paper/2309.01439/) (2023)

## Built on by (most cited first, 8 of 8)

- [Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training](https://mlsanity.com/paper/2411.16061/) (2024)
- [Autonomous Driving with Spiking Neural Networks](https://mlsanity.com/paper/2405.19687/) (2024)
- [STAA-SNN: Spatial-Temporal Attention Aggregator for Spiking Neural Networks](https://mlsanity.com/paper/2503.02689/) (2025)
- [Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networks](https://mlsanity.com/paper/2503.03144/) (2025)
- [DA-LIF: Dual Adaptive Leaky Integrate-and-Fire Model for Deep Spiking Neural Networks](https://mlsanity.com/paper/2502.10422/) (2025)
- [TS-SNN: Temporal Shift Module for Spiking Neural Networks](https://mlsanity.com/paper/2505.04165/) (2025)
- [Head-Tail-Aware KL Divergence in Knowledge Distillation for Spiking Neural Networks](https://mlsanity.com/paper/2504.20445/) (2025)
- [Learnable Burst Quantization for Expressive and Efficient Spiking Neural Networks](https://mlsanity.com/paper/2604.25688/) (2026)

## Abstract (arXiv)

The attention mechanism has been proven to be an effective way to improve spiking neural network (SNN). However, based on the fact that the current SNN input data flow is split into tensors to process on GPUs, none of the previous works consider the properties of tensors to implement an attention module. This inspires us to rethink current SNN from the perspective of tensor-relevant theories. Using tensor decomposition, we design the textitprojected full attention (PFA) module, which demonstrates excellent results with linearly growing parameters. Specifically, PFA is composed by the textitlinear projection of spike tensor (LPST) module and textitattention map composing (AMC) module. In LPST, we start by compressing the original spike tensor into three projected tensors using a single property-preserving strategy with learnable parameters for each dimension. Then, in AMC, we exploit the inverse procedure of the tensor decomposition process to combine the three tensors into the attention map using a so-called connecting factor. To validate the effectiveness of the proposed PFA module, we integrate it into the widely used VGG and ResNet architectures for classification tasks. Our method achieves state-of-the-art performance on both static and dynamic benchmark datasets, surpassing the existing SNN models with Transformer-based and CNN-based backbones.