# Softmax-Free Attention in Spiking Transformers as the Zero-Temperature Limit of Two-Level Systems

arXiv: https://arxiv.org/abs/2610.15968 · HTML: https://mlsanity.com/paper/2610.15968/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 11 Oct 2026.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [Dense Associative Memory for Pattern Recognition](https://mlsanity.com/paper/1606.01164/) (2016)
- [Spike-driven Transformer](https://mlsanity.com/paper/2307.01694/) (2023)
- [Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips](https://mlsanity.com/paper/2404.03663/) (2024)
- [SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks](https://mlsanity.com/paper/2302.13939/) (2023)
- [QKFormer: Hierarchical Spiking Transformer using Q-K Attention](https://mlsanity.com/paper/2403.16552/) (2024)

## Abstract (arXiv)

Spiking Transformers achieve energy efficiency by discarding softmax normalization in favor of thresholded, binary attention. However, this removal breaks the thermodynamic framework used to analyze attention networks, as the normalized measure and partition function cease to exist. We restore this theoretical foundation by proving that softmax-free spiking attention operates exactly as the zero-temperature limit of a product of independent two-level systems. This correspondence places spiking attention strictly in the frozen ordered phase of the thermodynamic phase diagram for all inputs. Consequently, the entropic cost of organizing attention collapses, replaced by a Landauer-type physical floor governed by action-potential generation. Furthermore, this ordered-phase pinning mathematically explains the optimization difficulty caused by vanishing surrogate gradients. We validate this theory with forward-relaxation experiments, placing both standard and neuromorphic attention mechanisms on a unified thermodynamic axis.