# Phase Space Attention:A Hairer Lift Circumvents the Single-Layer Induction Obstruction

arXiv: https://arxiv.org/abs/2609.32319 · HTML: https://mlsanity.com/paper/2609.32319/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 52 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 14 of 14)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Test-time regression: a unifying framework for designing sequence models with associative memory](https://mlsanity.com/paper/2501.12352/) (2025)
- [One-layer transformers fail to solve the induction heads task](https://mlsanity.com/paper/2408.14332/) (2024)
- [Critical attention scaling in long-context transformers](https://mlsanity.com/paper/2510.05554/) (2025)
- [Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation](https://mlsanity.com/paper/2505.24333/) (2025)
- [Differential Transformer](https://mlsanity.com/paper/2410.05258/) (2024)
- [KV Shifting Attention Enhances Language Modeling](https://mlsanity.com/paper/2411.19574/) (2024)
- [On the Power of Convolution Augmented Transformer](https://mlsanity.com/paper/2407.05591/) (2024)
- [DoPE: Denoising Rotary Position Embedding](https://mlsanity.com/paper/2511.09146/) (2025)
- [Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence](https://mlsanity.com/paper/2606.12058/) (2026)
- [Momentum Attention: The Physics of In-Context Learning and Spectral Forensics for Mechanistic Interpretability](https://mlsanity.com/paper/2602.04902/) (2026)
- [SympFormer: Accelerated attention blocks via Inertial Dynamics on Density Manifolds](https://mlsanity.com/paper/2603.16535/) (2026)
- [Invertible Query-Key Coupling Composes with Attention Mechanisms](https://mlsanity.com/paper/2604.01683/) (2026)

## Abstract (arXiv)

We circumvent the Sanford-Hsu-Telgarsky (SHT) single-layer induction obstruction within a linear, one-step, causal, bilinear, symplectically consistent design class on the post-RoPE substrate, by lifting attention onto a symplectic phase space, mirroring Hairer's lift of Stormer-Verlet. The lift exits the premise of the SHT counting argument rather than the bound itself. The reframing exhibits the obstruction as a filter-order gap: a one-layer bilinear score realises a z-transform of joint order (0,0), whereas the induction discriminator requires key-side order ≥ 1. Applying the symplectic upper shear Mγ:(q,p)mapsto(q+γ p,p) to the post-RoPE query and key streams closes it. We prove this lift is unique within the factorised subclass, exactly symplectic at operator level, and requires post-RoPE placement; and in an explicit T4-only Gaussian reduction we derive a closed-form two-branch induction phase transition, held out at r=0.9876 with zero fitted parameters. That law is an analytically solvable limit, not a robust prediction: restoring the T3 channel moves γc at dk=64 from 1.030 to 0.569 and removes the crossover. Deployability follows by exact derivation: the KV cache is unchanged, prefix reuse and speculative decoding are preserved, overhead is 6d FLOPs per token per layer, INT8 headroom grows by at most log2(1+2γ) bits, fused kernels are unmodified, and no parameters are added. At 91.3M parameters a supercritical sweep locates an emergence band: induction forms 3/3 seeds at γ=0.80 in a mean of 717 steps, against 2/3 seeds and 2700 steps at γ=0. Adverse results are reported as directly: a key-only half-lift reaches 0.949 against 0.811 for the symmetric operator, so if induction accuracy is the objective, the half-lift is the better construction. Forty-one notebooks and result files ship as ancillary material.