mlsanityData to 7 Oct 2026

Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story

arXiv 2511.15210 ↗First posted 19 Nov 2025cs.CL cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Intrinsic dimension (ID) is an important tool in modern LLM analysis, informing studies of training dynamics, scaling behavior, and dataset structure, yet its textual determinants remain underexplored. We provide the first comprehensive study grounding ID in interpretable text properties through cross-encoder analysis, linguistic features, and sparse autoencoders (SAEs). In this work, we establish three key findings. First, ID is complementary to entropy-based metrics: after controlling for length, the two are uncorrelated, with ID capturing geometric complexity orthogonal to prediction quality. Second, ID exhibits robust genre stratification: scientific prose shows low ID (~8), encyclopedic content medium ID (~9), and creative/opinion writing high ID (~10.5) across all models tested. This reveals that contemporary LLMs find scientific text "representationally simple" while fiction requires additional degrees of freedom. Third, using SAEs, we identify causal features: scientific signals (formal tone, report templates, statistics) reduce ID; humanized signals (personalization, emotion, narrative) increase it. Steering experiments confirm these effects are causal. Thus, for contemporary models, scientific writing appears comparatively "easy", whereas fiction, opinion, and affect add representational degrees of freedom. Our multi-faceted analysis provides practical guidance for the proper use of ID and the sound interpretation of ID-based results.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 27 of its 43 references that are in the map. First posted on arXiv 19 Nov 2025.

Builds on

27 of 43 references.

References · 27

  1. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  2. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  3. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  4. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  5. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  6. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  7. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  8. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
  9. 2024RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectorscited by 157
  10. 2023Intrinsic Dimension Estimation for Robust Detection of AI-Generated Textscited by 111
  11. 2021Intrinsic Dimension, Persistent Homology and Generalization in Neural Networkscited by 70
  12. 2024Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimensioncited by 45
15 more
  1. 2024Detection and Measurement of Syntactic Templates in Generated Textcited by 44
  2. 2023The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Modelscited by 37
  3. 2024Benchmarking Linguistic Diversity of Large Language Modelscited by 37
  4. 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Datacited by 30
  5. 2022Intrinsic Dimensionality Estimation within Tight Localities: A Theoretical and Experimental Analysiscited by 22
  6. 2023AI-generated text boundary detection with RoFTcited by 20
  7. 2024Robust AI-Generated Text Detection by Restricted Embeddingscited by 18
  8. 2025The Intrinsic Dimension of Prompts in Internal Representations of Large Language Modelscited by 17
  9. 2018Persistent Homology and the Upper Box Dimensioncited by 15
  10. 2024Geometric Signatures of Compositionality Across a Language Model's Lifetimecited by 14
  11. 2025Feature-Level Insights into Artificial Text Detection with Sparse Autoencoderscited by 14
  12. 2025Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddingscited by 10
  13. 2022NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimensioncited by 5
  14. 2025Geometric Metrics and LLMs: What They Measure and When They Workcited by 4
  15. 2025Memorization in Language Models through the Lens of Intrinsic Dimensioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Psychosis involves a deficit of information compression in connected speechcited by 0