From Direction to Magnitude: How Multimodal Instruction-Tuning Reorganizes the Geometric Encoding of Identity-Specifying Prompts in Transformer Hidden States
On the map: LLM reasoning → Prompting and chain-of-thought reasoning
Abstract (arXiv)
Versions 1 and 2 of this preprint reported that an identity-specifying system prompt leaves a geometric fingerprint in the final-layer hidden-state trajectories of four open-weight language models, and that instruction tuning moves this fingerprint from the direction to the magnitude of the hidden-state vector. An audit of their code and data found the following. The curvature statistic described as Ollivier-Ricci curvature on Euclidean k-NN graphs was a non-standard Forman-type edge statistic on graphs built from temporal and cosine k-NN edges. Its released test permuted pooled edges instead of trajectories, and the published p-values came from unreleased code. The quantity reported as the norm of the first generated state is the state at the last prompt position, from which the first output token is predicted. The generic control prompt was matched to the identity prompt in characters, not in tokens. This version corrects the methods, withdraws the regime-specific claims (one model per regime) and the direction-to-magnitude claim, and re-analyzes the data with added controls. What survives is narrower. Centroid distance, maximum mean discrepancy and a linear probe separate every pair of prompt conditions in every model, while the curvature statistic exceeds its split-half noise floor in only four of twelve comparisons. In Gemma-4-E4B-it this state has a lower norm under the identity prompt than under a token-length-matched generic prompt (138.1 vs. 216.5; Cohen's d = -5.45; n = 20). Its direction also separates the conditions, and the effect fits the state's role in planning the output: the identity prompt instructs a pause before every answer, and the model opens 98 of 100 responses with a pause marker. When the first token is fixed, the norm ordering reverses. The base model continues the prompt template instead of answering. A redesigned follow-up study is in preparation.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 5 of its 18 references that are in the map. First posted on arXiv 10 Jul 2026.
Builds on
5 of 18 references.
References · 5
- 2016Understanding intermediate layers using linear classifier probescited by 1,353
- 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
- 2026The Assistant Axis: Situating and Stabilizing the Default Persona of Language Modelscited by 90
- 2020Ollivier-Ricci curvature convergence in random geometric graphscited by 11
- 2023Continuum Limits of Ollivier's Ricci Curvature on data clouds: pointwise consistency and global lower boundscited by 6
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Trajectory Geometry of Transformer Representations Across Layers
- 2026The System Prompt Illusion: How Instruction Preambles Modify Computation in Language Models
- 2025Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
- 2026A Phenomenology of Neural Geometry in Vision-Language-Action Models
- 2025Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits
- 2026FishBack: Pullback Fisher Geometry for Optimal Activation Steering in Transformers
- 2026Contrastive Projection: Reading Transformer Internals by Differencing Logit Lenses
- 2025Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities