# The Routing and Filtering Structure of Attention

arXiv: https://arxiv.org/abs/2605.18826 · HTML: https://mlsanity.com/paper/2605.18826/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 7 references that are in the map.
- First posted on arXiv 12 May 2026.

## Builds on (most cited first, 6 of 6)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [RecurrentGemma: Moving Past Transformers for Efficient Open Language Models](https://mlsanity.com/paper/2404.07839/) (2024)

## Built on by (most cited first, 2 of 2)

- [Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention](https://mlsanity.com/paper/2607.06621/) (2026)
- [Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention](https://mlsanity.com/paper/2609.22005/) (2026)

## Abstract (arXiv)

The attention interaction matrix QKtop contains two entangled computations: a skew-symmetric component that redistributes information between positions (routing) and a symmetric component that scales mutual relevance (filtering). We decompose 1776 heads across five pretrained transformers and find routing operating at low rank, well below the routing capacity allocated by the weight kernel. We introduce S-D attention as a diagnostic parameterization that disentangles routing from filtering by construction with guaranteed stability (Re(λ) ≤ 0) and trains stably without layer normalization. When disentangled and unnormalized, routing self-organizes into a spectral cascade, effective rank 2 at the first layer, expanding with depth across six scales from 7M to 355M parameters. The cascade predicts where attention can be simplified: linearizing the first seven layers of 125M S-D attention costs <5% perplexity, whereas standard attention collapses under the same intervention. The linearizable region widens with depth. Replacing the first four layers with ELU+1 linear attention reaches within 1.4% of baseline at full head dimension. Cascade-allocated architectures trade attention parameters for perplexity (47%-65% fewer attention parameters at +3.9% to +8.4% PPL). The routing-filtering decomposition makes the spectral budget legible; the cascade makes it actionable.