# Emergent Causal-Geometric Dynamics Across Depth in Large Language Models

arXiv: https://arxiv.org/abs/2602.04931 · HTML: https://mlsanity.com/paper/2602.04931/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 4 Feb 2026.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?](https://mlsanity.com/paper/2005.00719/) (2020)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [Transformers need glasses! Information over-squashing in language tasks](https://mlsanity.com/paper/2406.04267/) (2024)
- [Emergence of a High-Dimensional Abstraction Phase in Language Transformers](https://mlsanity.com/paper/2405.15471/) (2024)
- [Angular Steering: Behavior Control via Rotation in Activation Space](https://mlsanity.com/paper/2510.26243/) (2025)
- [The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models](https://mlsanity.com/paper/2311.05928/) (2023)
- [A polar coordinate system represents syntax in large language models](https://mlsanity.com/paper/2412.05571/) (2024)
- [Characterizing Large Language Model Geometry Helps Solve Toxicity Detection and Generation](https://mlsanity.com/paper/2312.01648/) (2023)
- [From Directions to Regions: Decomposing Activations in Language Models via Local Geometry](https://mlsanity.com/paper/2602.02464/) (2026)
- [From Directions to Cones: Exploring Multidimensional Representations of Propositional Facts in LLMs](https://mlsanity.com/paper/2505.21800/) (2025)
- [A Survey on Training-free Alignment of Large Language Models](https://mlsanity.com/paper/2508.09016/) (2025)

## Abstract (arXiv)

Geometric analyses of large language model (LLM) representations reveal structured variation across depth but remain fundamentally correlational with respect to token prediction formation. Meanwhile, causal interventions expose depth-dependent efficacy profiles without a unifying account of their representational dynamics. A complete account of LLM function requires explaining how representational structure evolves across depth to causally produce predictions. We synthesize these perspectives by combining geometric analysis with mechanistic interventions, explicitly centralizing depth-wise dynamics as the organizing axis for interpreting LLM function. In decoder-only LLMs, we identify a sharp transition from context-processing to prediction-forming computation, accompanied by a more gradual reorganization of representational geometry across layers. This synthesis reveals a late-layer geometric code in which angular structure parameterizes next-token distributional similarity and enables selective causal control over predictions, while representation norms encode information largely decoupled from prediction. Together, our results provide a synthesis of causal and geometric perspectives, yielding a mechanistic account of how control-relevant geometric dynamics across depth transform context into prediction in language models. This perspective reconciles previously puzzling findings and implies that layer-wise function cannot be understood or effectively intervened upon in isolation, but only within the emergent global dynamical structure of the network.