mlsanityData to 14 Oct 2026

Transformer Field Theory: A Response-Theoretic Approach to Mechanistic Interpretability

arXiv 2605.25225 ↗First posted 24 May 2026cs.LG cs.AIcited by 0

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

Mechanistic interpretability often studies Transformer behavior by intervening on internal activations through activation patching, causal tracing, path patching, and steering directions. This paper develops Transformer Field Theory: a response-theoretic framework in which the residual stream of a fixed forward pass is treated as a Transformer field over layer depth and token position. In this formulation, patching becomes a localized source insertion into the Transformer field, first-order sensitivity fields predict patch effects, Green functions describe downstream propagation, and patch selection is posed as an adjoint inverse problem. Empirically, we test the theory's forward response objects in GPT-2-style autoregressive Transformers. Localized Transformer-field interventions exhibit a bounded local linear regime; first-order sensitivities predict patch effects across layer-token sites; localized sources generate structured anisotropic Transformer-field propagation; high-sensitivity sites and sliced Green operators provide reduced response descriptions; and prompt-induced Transformer-field displacements partially transfer answer behavior. These results establish sensitivities, Transformer-field responses, and sliced Green operators as practical objects for organizing patching experiments, while providing the forward mathematical basis for patch-site inference and cross-scale response transfer.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 15 of its 29 references that are in the map. First posted on arXiv 24 May 2026.

Builds on

15 of 29 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Attribution Patching Outperforms Automated Circuit Discovery16 Oct
  2. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small1 Nov

Other references · 13

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018Neural Ordinary Differential Equationscited by 3,779
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  5. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  6. 2023Steering Language Models With Activation Engineeringcited by 922
  7. 2024Scaling and evaluating sparse autoencoderscited by 553
  8. 2023In-Context Learning Creates Task Vectorscited by 312
  9. 2023Function Vectors in Large Language Modelscited by 308
  10. 2014Understanding image representations by measuring their equivariance and equivalencecited by 243
  11. 2021Revisiting Model Stitching to Compare Neural Representationscited by 209
  12. 2023Localizing Model Behavior with Path Patchingcited by 195
1 more
  1. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 111

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Automated Circuit Discovery for Mechanistic Interpretability
  2. 2025Decomposition of Small Transformer Models
  3. 2025Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
  4. 2025nnterp: A Standardized Interface for Mechanistic Interpretability of Transformers
  5. 2026Geometric and Behavioral Stratification in Transformer Residual Streams
  6. 2025Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits
  7. 2026Variance Is Not Importance: Structural Analysis of Transformer Compressibility Across Model Scales
  8. 2026LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference