SpiLiFormer: Enhancing Spiking Transformers with Lateral Inhibition
Not on the map. Why?
Abstract (arXiv)
Spiking Neural Networks (SNNs) based on Transformers have garnered significant attention due to their superior performance and high energy efficiency. However, the spiking attention modules of most existing Transformer-based SNNs are adapted from those of analog Transformers, failing to fully address the issue of over-allocating attention to irrelevant contexts. To fix this fundamental yet overlooked issue, we propose a Lateral Inhibition-inspired Spiking Transformer (SpiLiFormer). It emulates the brain's lateral inhibition mechanism, guiding the model to enhance attention to relevant tokens while suppressing attention to irrelevant ones. Our model achieves state-of-the-art (SOTA) performance across multiple datasets, including CIFAR-10 (+0.45%), CIFAR-100 (+0.48%), CIFAR10-DVS (+2.70%), N-Caltech101 (+1.94%), and ImageNet-1K (+1.6%). Notably, on the ImageNet-1K dataset, SpiLiFormer (69.9M parameters, 4 time steps, 384 resolution) outperforms E-SpikeFormer (173.0M parameters, 8 time steps, 384 resolution), a SOTA spiking Transformer, by 0.46% using only 39% of the parameters and half the time steps. The code and model checkpoints are publicly available at https://github.com/KirinZheng/SpiLiFormer.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 32 of its 51 references that are in the map. First posted on arXiv 20 Mar 2025.
Builds on
32 of 51 references.
References · 32
- 2017Attention Is All You Needcited by 60,361
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
- 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
- 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
- 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
- 2017Random Erasing Data Augmentationcited by 1,239
- 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
20 more
- 2016Deep Networks with Stochastic Depthcited by 979
- 2021Deep Residual Learning in Spiking Neural Networkscited by 331
- 2015Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccadescited by 314
- 2022Spikformer: When Spiking Neural Network Meets Transformercited by 271
- 2022Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingcited by 198
- 2019Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detectioncited by 177
- 2023Spike-driven Transformercited by 146
- 2024Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chipscited by 103
- 2023SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networkscited by 88
- 2024Integer-Valued Training and Spike-Driven Inference Spiking Neural Network for High-performance and Energy-efficient Object Detectioncited by 67
- 2021HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noisecited by 65
- 2024Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Trainingcited by 59
- 2024QKFormer: Hierarchical Spiking Transformer using Q-K Attentioncited by 49
- 2022Optimizing Relevance Maps of Vision Transformers Improves Robustnesscited by 41
- 2022Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformerscited by 30
- 2024Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticketcited by 30
- 2024Spiking Transformer with Spatial-Temporal Attentioncited by 19
- 2024Selective Attention: Enhancing Transformer through Principled Context Controlcited by 16
- 2022Spiking Neural Networks for event-based action recognition: A new task to understand their advantagecited by 9
- 2023Spiking-LEAF: A Learnable Auditory front-end for Spiking Neural Networkscited by 6
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.