# Scale Determines Whether Language Models Organize Representation Geometry for Prediction

arXiv: https://arxiv.org/abs/2605.17084 · HTML: https://mlsanity.com/paper/2605.17084/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 16 May 2026.

## Builds on (most cited first, 28 of 28)

- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Revisiting Model Stitching to Compare Neural Representations](https://mlsanity.com/paper/2106.07682/) (2021)
- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [Analyzing Transformers in Embedding Space](https://mlsanity.com/paper/2209.02535/) (2022)
- [The geometry of hidden representations of large transformer models](https://mlsanity.com/paper/2302.00294/) (2023)
- [Transformers represent belief state geometry in their residual stream](https://mlsanity.com/paper/2405.15943/) (2024)
- [Spectral Filters, Dark Signals, and Attention Sinks](https://mlsanity.com/paper/2402.09221/) (2024)
- [Shared Global and Local Geometry of Language Model Embeddings](https://mlsanity.com/paper/2503.21073/) (2025)
- [Tracing the Representation Geometry of Language Models from Pretraining to Post-training](https://mlsanity.com/paper/2509.23024/) (2025)
- [Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations](https://mlsanity.com/paper/2408.15417/) (2024)
- [Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning](https://mlsanity.com/paper/2411.02344/) (2024)
- [Why Linear Interpretability Works: Invariant Subspaces as a Result of Architectural Constraints](https://mlsanity.com/paper/2602.09783/) (2026)
- [Disentangling Geometry, Performance, and Training in Language Models](https://mlsanity.com/paper/2602.20433/) (2026)

## Built on by (most cited first, 2 of 2)

- [Decodable In-Context State and Model Output Across Training](https://mlsanity.com/paper/2609.31401/) (2026)
- [The Geometry of Inference in Transformer Residual Streams](https://mlsanity.com/paper/2609.37824/) (2026)

## Abstract (arXiv)

In language models, what a representation encodes is determined by the geometry of its representation space: distances, not activations, carry meaning. Existing tools characterize the shape of this geometry but do not ask what that shape is organized for. We introduce Subspace PGA, a metric that tests whether a layer's distance structure aligns with the readout subspace of the unembedding matrix WU more than with random subspaces of equal size. Across seven Pythia models (70M--6.9B) and three cross-family models, intermediate geometry is significantly organized for prediction (peak z = 9--24), but the degree is scale-dependent: small models (d ≤ 1024) progressively lose it at late layers during training -- even as loss keeps improving -- while large models (d ≥ 2048) preserve it throughout. We trace this to a capacity trade-off: a few dominant directions migrate away from WU's readout, masking rather than destroying the predictive structure beneath, and removing them restores alignment. Neither spectral metrics nor loss curves capture this distinction. Scale thus determines not only how well a model predicts, but how its representation geometry is organized to do so.