# Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension

arXiv: https://arxiv.org/abs/2604.15769 · HTML: https://mlsanity.com/paper/2604.15769/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 17 Apr 2026.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Are Transformers universal approximators of sequence-to-sequence functions?](https://mlsanity.com/paper/1912.10077/) (2019)
- [Theoretical Limitations of Self-Attention in Neural Sequence Models](https://mlsanity.com/paper/1906.06755/) (2019)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [The Parallelism Tradeoff: Limitations of Log-Precision Transformers](https://mlsanity.com/paper/2207.00729/) (2022)
- [SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence](https://mlsanity.com/paper/2310.16620/) (2023)
- [Spike-driven Transformer](https://mlsanity.com/paper/2307.01694/) (2023)
- [Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips](https://mlsanity.com/paper/2404.03663/) (2024)
- [SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks](https://mlsanity.com/paper/2302.13939/) (2023)
- [Tighter Bounds on the Expressivity of Transformer Encoders](https://mlsanity.com/paper/2301.10743/) (2023)
- [QKFormer: Hierarchical Spiking Transformer using Q-K Attention](https://mlsanity.com/paper/2403.16552/) (2024)
- [SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation](https://mlsanity.com/paper/2308.10873/) (2023)
- [SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks](https://mlsanity.com/paper/2403.14302/) (2024)
- [Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticket](https://mlsanity.com/paper/2401.02020/) (2024)
- [On the Universal Approximability and Complexity Bounds of Quantized ReLU Neural Networks](https://mlsanity.com/paper/1802.03646/) (2018)
- [On the Intrinsic Structures of Spiking Neural Networks](https://mlsanity.com/paper/2207.04876/) (2022)
- [Expressivity of Spiking Neural Networks](https://mlsanity.com/paper/2308.08218/) (2023)
- [Universal Approximation Theorems of Fully Connected Binarized Neural Networks](https://mlsanity.com/paper/2102.02631/) (2021)

## Abstract (arXiv)

Spiking transformers achieve competitive accuracy with conventional transformers while offering 38-57× energy efficiency on neuromorphic hardware, yet no theoretical framework guides their design. This paper establishes the first comprehensive expressivity theory for spiking self-attention. We prove that spiking attention with Leaky Integrate-and-Fire neurons is a universal approximator of continuous permutation-equivariant functions, providing explicit spike circuit constructions including a novel lateral inhibition network for softmax normalization with proven O(1/√T) convergence. We derive tight spike-count lower bounds via rate-distortion theory: varepsilon-approximation requires Omega(Lf2 nd/varepsilon2) spikes, with rigorous information-theoretic derivation. Our key insight is input-dependent bounds using measured effective dimensions (deff=47--89 for CIFAR/ImageNet), explaining why T=4 timesteps suffice despite worst-case T ≥ 10,000 predictions. We provide concrete design rules with calibrated constants (C=2.3, 95% CI: [1.9, 2.7]). Experiments on Spikformer, QKFormer, and SpikingResformer across vision and language benchmarks validate predictions with R2=0.97 (p<0.001). Our framework provides the first principled foundation for neuromorphic transformer design.