mlsanityData to 7 Oct 2026

Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras

arXiv 2609.01129 ↗First posted 1 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We identify a recurrent algebraic regularity in Transformer attention: a sparse subset of effective OV operators T=OV^top nearly closes under composition, T2≈α T. Across six pretrained endpoints spanning 2.8B--235B parameters, 3.98--8.00% of heads reach squared closure alignment P≥0.9, while no matched within-layer O/V mismatch does. An exact principal-coordinate factorization, T=QOKQV^top and T2=QO(KDK)QV^top, separates within-support transport from read--write return geometry. Across all 7,304 heads in nine MHA/GQA models, scrambling only the orientation of K while preserving singular values, norms, factor spans, and principal angles reduces median closure from 0.336 to 1.04×10-4; trained orientation wins for 98.64% of heads and in every layer. Constructive searches show that high closure is feasible in every surveyed layer, but usually not attained. Retrospective trajectories in three independently trained lineages further separate broadly available capacity from the orientations attained by final strong heads. Under exact value sharing, headwise closure extends to a right-action algebra, TiTj=αjTi. Seven-model experiments verify the approximate law and reveal distinct oblique projections with a shared value-defined kernel. These results characterize scaled idempotence as a sparse trained orientation within broadly available geometric capacity and show how value sharing extends a headwise relation into a local operator algebra.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 20 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

14 of 20 references.

References · 14

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2023Mistral 7Bcited by 3,065
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  9. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  10. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  11. 2024Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learningcited by 6
  12. 2026Measuring Affinity between Attention-Head Weight Subspaces via the Projection Kernelcited by 5
2 more
  1. 2026Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuitscited by 1
  2. 2026The Communication Map of a Transformercited by 1

Built on by

Not cited yet.