SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models
Not on the map. Why?
Abstract (arXiv)
Residual-stream analysis asks how language-model computation evolves across depth, but intermediate decoding requires comparable readout coordinates across layers. If embedding anchors and unembedding readout disagree on the chosen span, apparent motion may reflect measurement drift rather than computation. We introduce Semantic Reference Frames (SemRF), an anchor-based formalism separating semantic measurement from residual dynamics. A SemRF fixes anchors and measures states against them. Pseudo-inverse tying gives exact synchronization; under restricted bi-invertibility, SemRF yields stable semantic-basis coordinates, distortion bounds, and near-identity changes. With the frame fixed, residual computation becomes a depthwise semantic trajectory. The anchors induce a semantic Voronoi diagram: distance, or evidence such as logits, assigns each layer to a coarse cell, while coordinates retain within-cell motion and margins. We define layerwise steps, contribution profiles, and imbalance diagnostics, then use the Voronoi trace to define a margin-relaxed tube. The canonical trace is the minimum-action path inside this tube; when nonempty with positive quadratic weight, it is unique and obeys a discrete spline equation away from active constraints. Excess action controls step, curvature, and profile mismatch. Low curvature implies piecewise-linear compressibility and local knowledge density: lower trace complexity means fewer semantic knots. Through the parameter-to-trajectory map, this gives a conditional link to parameter efficiency: among admissible settings fitting data, lower-action and lower-complexity traces use fewer semantic degrees of freedom. The guarantees require controlled interface error and small projection residual under explicit tube constraints.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 28 references that are in the map. First posted on arXiv 30 Jun 2026.
Builds on
24 of 28 references.
References · 24
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2019Language Models as Knowledge Bases?cited by 1,922
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
- 2022Toy Models of Superpositioncited by 928
- 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
- 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
- 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
- 2019Visualizing and Measuring the Geometry of BERTcited by 215
- 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 175
12 more
- 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
- 2024Transformers represent belief state geometry in their residual streamcited by 73
- 2024Densing Law of LLMscited by 39
- 2021Learning to Remove: Towards Isotropic Pre-trained BERT Embeddingcited by 23
- 2024Residual Stream Analysis with Multi-Layer SAEscited by 18
- 2024Vocabulary-Defined Semantics: Latent Space Clustering for Improving In-Context Learningcited by 7
- 2026Semantic Tube Prediction: Beating LLM Data Efficiency with JEPAcited by 6
- 2024A Semantic-Aware Layer-Freezing Approach to Computation-Efficient Fine-Tuning of Language Modelscited by 5
- 2025Equivalence of Context and Parameter Updates in Modern Transformer Blockscited by 4
- 2026The Representational Geometry of Numbercited by 4
- 2024Characterizing stable regions in the residual stream of LLMscited by 3
- 2026Rethinking Weight Tying: Pseudo-Inverse Tying for LM Stable Training and Updatescited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026The Geometry of Inference in Transformer Residual Streams29 Sep 2026