# Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations

arXiv: https://arxiv.org/abs/2601.03779 · HTML: https://mlsanity.com/paper/2601.03779/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 65 references that are in the map.
- First posted on arXiv 7 Jan 2026.

## Builds on (most cited first, 30 of 30)

- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [Estimating the intrinsic dimension of datasets by a minimal neighborhood information](https://mlsanity.com/paper/1803.06992/) (2018)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs](https://mlsanity.com/paper/2309.07311/) (2023)
- [The geometry of hidden representations of large transformer models](https://mlsanity.com/paper/2302.00294/) (2023)
- [Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts](https://mlsanity.com/paper/2306.04723/) (2023)
- [Syntactic Structure from Deep Learning](https://mlsanity.com/paper/2004.10827/) (2020)
- [Emergence of a High-Dimensional Abstraction Phase in Language Transformers](https://mlsanity.com/paper/2405.15471/) (2024)
- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [Do Language Models Use Their Depth Efficiently?](https://mlsanity.com/paper/2505.13898/) (2025)
- [Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension](https://mlsanity.com/paper/2402.18048/) (2024)
- [Bridging Information-Theoretic and Geometric Compression in Language Models](https://mlsanity.com/paper/2310.13620/) (2023)
- [Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models using Minimal Pairs](https://mlsanity.com/paper/2403.17299/) (2024)
- [Ranking the information content of distance measures](https://mlsanity.com/paper/2104.15079/) (2021)
- [Geometric Signatures of Compositionality Across a Language Model's Lifetime](https://mlsanity.com/paper/2410.01444/) (2024)
- [The representation landscape of few-shot learning and fine-tuning in large language models](https://mlsanity.com/paper/2409.03662/) (2024)
- [Characterizing Large Language Model Geometry Helps Solve Toxicity Detection and Generation](https://mlsanity.com/paper/2312.01648/) (2023)
- [A quantitative analysis of semantic information in deep representations of text and images](https://mlsanity.com/paper/2505.17101/) (2025)
- [Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations](https://mlsanity.com/paper/2506.16678/) (2025)
- [Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language Models](https://mlsanity.com/paper/2506.02132/) (2025)
- [Not a nuisance but a useful heuristic: Outlier dimensions favor frequent tokens in language models](https://mlsanity.com/paper/2503.21718/) (2025)

## Built on by (most cited first, 3 of 3)

- [Abstraction Induces the Brain Alignment of Language and Speech Models](https://mlsanity.com/paper/2602.04081/) (2026)
- [Light or Full Verb? A Minimal-Pair Dataset for Probing Phraseological Competence in Language Models](https://mlsanity.com/paper/2606.05087/) (2026)
- [Semantic Register Compression in Multi-Agent LLM Cascades](https://mlsanity.com/paper/2607.14119/) (2026)

## Abstract (arXiv)

We explore intrinsic dimension (ID) of LLM representations as a marker of linguistic complexity. Specifically, we test whether ID differences across model layers reflect well-known complexity contrasts established in (psycho)linguistics: coordination vs. subordination, right-branching vs. center-embedding, and unambiguous vs. ambiguous attachment. Our results on six different LLMs show that these contrasts are consistently reflected in ID differences, with more complex phenomena eliciting higher ID profiles. Notably, ID differences emerge at different points across layers for different contrasts, also reaching their peaks at different stages. Further experiments using representational similarity and layer pruning confirm the trends. We conclude that ID is a useful marker of linguistic complexity in LLMs, that it points to similar linguistic processing steps across disparate LLMs, and that it has the potential to differentiate between different types of complexity.