# Testing the spin-bath view of self-attention: A Hamiltonian analysis of GPT-2 Transformer

arXiv: https://arxiv.org/abs/2507.00683 · HTML: https://mlsanity.com/paper/2507.00683/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 0 of its 4 references that are in the map.
- First posted on arXiv 1 Jul 2025.

## Built on by (most cited first, 2 of 2)

- [Many-body Tipping Dynamics of ChatGPT-like AIs](https://mlsanity.com/paper/2607.25279/) (2026)
- [Interaction Creates Dynamical AI Behavior Absent in Isolation](https://mlsanity.com/paper/2608.07457/) (2026)

## Abstract (arXiv)

The recently proposed physics-based framework by Huo and Johnson~citehuo2024capturing models the attention mechanism of Large Language Models (LLMs) as an interacting two-body spin system, offering a first-principles explanation for phenomena like repetition and bias. Building on this hypothesis, we extract the complete Query-Key weight matrices from a production-grade GPT-2 model and derive the corresponding effective Hamiltonian for every attention head. From these Hamiltonians, we obtain analytic phase boundaries and logit gap criteria that predict which token should dominate the next-token distribution for a given context. A systematic evaluation on 144 heads across 20 factual-recall prompts reveals a strong negative correlation between the theoretical logit gaps and the model's empirical token rankings (r≈-0.70, p<10-3).Targeted ablations further show that suppressing the heads most aligned with the spin-bath predictions induces the anticipated shifts in output probabilities, confirming a causal link rather than a coincidental association. Taken together, our findings provide the first strong empirical evidence for the spin-bath analogy in a production-grade model. In this work, we utilize the context-field lens, which provides physics-grounded interpretability and motivates the development of novel generative models bridging theoretical condensed matter physics and artificial intelligence.