Phase Space Attention:A Hairer Lift Circumvents the Single-Layer Induction Obstruction
Not on the map. Why?
Abstract (arXiv)
We circumvent the Sanford-Hsu-Telgarsky (SHT) single-layer induction obstruction within a linear, one-step, causal, bilinear, symplectically consistent design class on the post-RoPE substrate, by lifting attention onto a symplectic phase space, mirroring Hairer's lift of Stormer-Verlet. The lift exits the premise of the SHT counting argument rather than the bound itself. The reframing exhibits the obstruction as a filter-order gap: a one-layer bilinear score realises a z-transform of joint order (0,0), whereas the induction discriminator requires key-side order ≥ 1. Applying the symplectic upper shear Mγ:(q,p)mapsto(q+γ p,p) to the post-RoPE query and key streams closes it. We prove this lift is unique within the factorised subclass, exactly symplectic at operator level, and requires post-RoPE placement; and in an explicit T4-only Gaussian reduction we derive a closed-form two-branch induction phase transition, held out at r=0.9876 with zero fitted parameters. That law is an analytically solvable limit, not a robust prediction: restoring the T3 channel moves γc at dk=64 from 1.030 to 0.569 and removes the crossover. Deployability follows by exact derivation: the KV cache is unchanged, prefix reuse and speculative decoding are preserved, overhead is 6d FLOPs per token per layer, INT8 headroom grows by at most log2(1+2γ) bits, fused kernels are unmodified, and no parameters are added. At 91.3M parameters a supercritical sweep locates an emergence band: induction forms 3/3 seeds at γ=0.80 in a mean of 717 steps, against 2/3 seeds and 2700 steps at γ=0. Adverse results are reported as directly: a key-only half-lift reaches 0.949 against 0.811 for the symmetric operator, so if induction accuracy is the objective, the half-lift is the better construction. Forty-one notebooks and result files ship as ancillary material.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 52 references that are in the map. First posted on arXiv 26 Sep 2026.
Builds on
14 of 52 references.
References · 14
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
- 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
- 2024One-layer transformers fail to solve the induction heads taskcited by 32
- 2025Critical attention scaling in long-context transformerscited by 20
- 2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 17
- 2024Differential Transformercited by 13
- 2024KV Shifting Attention Enhances Language Modelingcited by 8
- 2024On the Power of Convolution Augmented Transformercited by 6
- 2025DoPE: Denoising Rotary Position Embeddingcited by 5
- 2026Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergencecited by 3
- 2026Momentum Attention: The Physics of In-Context Learning and Spectral Forensics for Mechanistic Interpretabilitycited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Latent Object Permanence: Topological Phase Transitions, Free-Energy Principles, and Renormalization Group Flows in Deep Transformer Manifolds
- 2026Dynamical phase selection controls compute scaling in looped transformers
- 2025Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Embedding
- 2026Beyond Diagonal State Space Models: Exact Non-Abelian Group Tracking, Solvability Barriers, and Geometric Physical Manifolds
- 2026Solver Agent: an Agentic AI Framework for Theoretical Physics Computations Applied to F-theory Uplifts of O3-planes and S-folds
- 2025Using physics-inspired Singular Learning Theory to understand grokking & other phase transitions in modern neural networks
- 2026The Geometry of Thought: Disclosing the Transformer as a Tropical Polynomial Circuit
- 2025Learning Physical Systems: Symplectification via Gauge Fixing in Dirac Structures