# Momentum Attention: The Physics of In-Context Learning and Spectral Forensics for Mechanistic Interpretability

arXiv: https://arxiv.org/abs/2602.04902 · HTML: https://mlsanity.com/paper/2602.04902/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability.
- First posted on arXiv 3 Feb 2026.

## Placed in

- Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/)

## Built on by (most cited first, 1 of 1)

- [Phase Space Attention:A Hairer Lift Circumvents the Single-Layer Induction Obstruction](https://mlsanity.com/paper/2609.32319/) (2026)

## Abstract (arXiv)

The Mechanistic Interpretability (MI) program has mapped the Transformer as a precise computational graph. We extend this graph with a conservation law and time-varying AC dynamics, viewing it as a physical circuit. We introduce Momentum Attention, a symplectic augmentation embedding physical priors via the kinematic difference operator pt = qt - qt-1, implementing the symplectic shear hatqt = qt + γ pt on queries and keys. We identify a fundamental Symplectic-Filter Duality: the physical shear is mathematically equivalent to a High-Pass Filter. This duality is our cornerstone contribution -- by injecting kinematic momentum, we sidestep the topological depth constraint (L ≥ 2) for induction head formation. While standard architectures require two layers for induction from static positions, our extension grants direct access to velocity, enabling Single-Layer Induction and Spectral Forensics via Bode Plots. We formalize an Orthogonality Theorem proving that DC (semantic) and AC (mechanistic) signals segregate into orthogonal frequency bands when Low-Pass RoPE interacts with High-Pass Momentum. Validated through 5,100+ controlled experiments (documented in Supplementary Appendices A--R and 27 Jupyter notebooks), our 125M Momentum model exceeds expectations on induction-heavy tasks while tracking a 350M baseline within ~2.9% validation loss. Dedicated associative recall experiments reveal a scaling law γ^* = 4.17 × N-0.74 establishing momentum-depth fungibility. We offer this framework as a complementary analytical toolkit connecting Generative AI, Hamiltonian Physics, and Signal Processing.