# Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story

arXiv: https://arxiv.org/abs/2511.15210 · HTML: https://mlsanity.com/paper/2511.15210/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 19 Nov 2025.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 27 of 27)

- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [The Intrinsic Dimension of Images and Its Impact on Learning](https://mlsanity.com/paper/2104.08894/) (2021)
- [Estimating the intrinsic dimension of datasets by a minimal neighborhood information](https://mlsanity.com/paper/1803.06992/) (2018)
- [RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors](https://mlsanity.com/paper/2405.07940/) (2024)
- [Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts](https://mlsanity.com/paper/2306.04723/) (2023)
- [Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks](https://mlsanity.com/paper/2111.13171/) (2021)
- [Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension](https://mlsanity.com/paper/2402.18048/) (2024)
- [Detection and Measurement of Syntactic Templates in Generated Text](https://mlsanity.com/paper/2407.00211/) (2024)
- [The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models](https://mlsanity.com/paper/2311.05928/) (2023)
- [Benchmarking Linguistic Diversity of Large Language Models](https://mlsanity.com/paper/2412.10271/) (2024)
- [Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data](https://mlsanity.com/paper/2411.06646/) (2024)
- [Intrinsic Dimensionality Estimation within Tight Localities: A Theoretical and Experimental Analysis](https://mlsanity.com/paper/2209.14475/) (2022)
- [AI-generated text boundary detection with RoFT](https://mlsanity.com/paper/2311.08349/) (2023)
- [Robust AI-Generated Text Detection by Restricted Embeddings](https://mlsanity.com/paper/2410.08113/) (2024)
- [The Intrinsic Dimension of Prompts in Internal Representations of Large Language Models](https://mlsanity.com/paper/2501.10573/) (2025)
- [Geometric Signatures of Compositionality Across a Language Model's Lifetime](https://mlsanity.com/paper/2410.01444/) (2024)
- [Persistent Homology and the Upper Box Dimension](https://mlsanity.com/paper/1802.00533/) (2018)
- [Feature-Level Insights into Artificial Text Detection with Sparse Autoencoders](https://mlsanity.com/paper/2503.03601/) (2025)
- [Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings](https://mlsanity.com/paper/2506.01435/) (2025)
- [NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension](https://mlsanity.com/paper/2211.12759/) (2022)
- [Geometric Metrics and LLMs: What They Measure and When They Work](https://mlsanity.com/paper/2509.25359/) (2025)
- [Memorization in Language Models through the Lens of Intrinsic Dimension](https://mlsanity.com/paper/2506.09591/) (2025)

## Built on by (most cited first, 1 of 1)

- [Psychosis involves a deficit of information compression in connected speech](https://mlsanity.com/paper/2609.15522/) (2026)

## Abstract (arXiv)

Intrinsic dimension (ID) is an important tool in modern LLM analysis, informing studies of training dynamics, scaling behavior, and dataset structure, yet its textual determinants remain underexplored. We provide the first comprehensive study grounding ID in interpretable text properties through cross-encoder analysis, linguistic features, and sparse autoencoders (SAEs). In this work, we establish three key findings. First, ID is complementary to entropy-based metrics: after controlling for length, the two are uncorrelated, with ID capturing geometric complexity orthogonal to prediction quality. Second, ID exhibits robust genre stratification: scientific prose shows low ID (~8), encyclopedic content medium ID (~9), and creative/opinion writing high ID (~10.5) across all models tested. This reveals that contemporary LLMs find scientific text "representationally simple" while fiction requires additional degrees of freedom. Third, using SAEs, we identify causal features: scientific signals (formal tone, report templates, statistics) reduce ID; humanized signals (personalization, emotion, narrative) increase it. Steering experiments confirm these effects are causal. Thus, for contemporary models, scientific writing appears comparatively "easy", whereas fiction, opinion, and affect add representational degrees of freedom. Our multi-faceted analysis provides practical guidance for the proper use of ID and the sound interpretation of ID-based results.