# Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras

arXiv: https://arxiv.org/abs/2609.01129 · HTML: https://mlsanity.com/paper/2609.01129/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 20 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Builds on (most cited first, 14 of 14)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning](https://mlsanity.com/paper/2411.02199/) (2024)
- [Measuring Affinity between Attention-Head Weight Subspaces via the Projection Kernel](https://mlsanity.com/paper/2601.10266/) (2026)
- [Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits](https://mlsanity.com/paper/2607.07066/) (2026)
- [The Communication Map of a Transformer](https://mlsanity.com/paper/2608.22007/) (2026)

## Abstract (arXiv)

We identify a recurrent algebraic regularity in Transformer attention: a sparse subset of effective OV operators T=OV^top nearly closes under composition, T2≈α T. Across six pretrained endpoints spanning 2.8B--235B parameters, 3.98--8.00% of heads reach squared closure alignment P≥0.9, while no matched within-layer O/V mismatch does. An exact principal-coordinate factorization, T=QOKQV^top and T2=QO(KDK)QV^top, separates within-support transport from read--write return geometry. Across all 7,304 heads in nine MHA/GQA models, scrambling only the orientation of K while preserving singular values, norms, factor spans, and principal angles reduces median closure from 0.336 to 1.04×10-4; trained orientation wins for 98.64% of heads and in every layer. Constructive searches show that high closure is feasible in every surveyed layer, but usually not attained. Retrospective trajectories in three independently trained lineages further separate broadly available capacity from the orientations attained by final strong heads. Under exact value sharing, headwise closure extends to a right-action algebra, TiTj=αjTi. Seven-model experiments verify the approximate law and reveal distinct oblique projections with a shared value-defined kernel. These results characterize scaled idempotence as a sparse trained orientation within broadly available geometric capacity and show how value sharing extends a headwise relation into a local operator algebra.