# Spiking Sequence Machines and Transformers

arXiv: https://arxiv.org/abs/2605.00662 · HTML: https://mlsanity.com/paper/2605.00662/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 1 May 2026.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [Deep Information Propagation](https://mlsanity.com/paper/1611.01232/) (2016)
- [Mean Field Residual Networks: On the Edge of Chaos](https://mlsanity.com/paper/1712.08969/) (2017)
- [Efficient spike encoding algorithms for neuromorphic speech recognition](https://mlsanity.com/paper/2207.07073/) (2022)

## Abstract (arXiv)

Sequence learning reduces to similarity-based retrieval over a temporally indexed representation space, a constraint on any sequence model, not a property of a specific architecture. We show that a spiking Sparse Distributed Memory sequence machine (2007) and the transformer (2017) independently instantiate the same five functional operations (encoding, context maintenance, associative retrieval, storage, and decoding), with cosine similarity as the shared retrieval primitive in both. We formalise a Phase-Latency Isomorphism showing that sinusoidal positional phase and spike timing are linearly related, and prove that dot product attention is invariant to this mapping up to a global scale factor on the positional component (Lemma 1). Empirically, frequency-compressed positional encoding fails to converge on a positionally demanding copy task, while a learned rank-based embedding matches or exceeds sinusoidal encoding, indicating that the critical property for positional representation is distance discriminability under dot-product similarity, not sinusoidal form. Time, phase, and rank are three instantiations of the same computational primitive, an ordered index whose structure survives similarity-based retrieval.