# The Intrinsic Dimension of Prompts in Internal Representations of Large Language Models

arXiv: https://arxiv.org/abs/2501.10573 · HTML: https://mlsanity.com/paper/2501.10573/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Steering and probing for LLM safety, one of 13 directions in Mechanistic interpretability.
- Builds on 41 of its 60 references that are in the map.
- First posted on arXiv 17 Jan 2025.

## Placed in

- Mechanistic interpretability / [Steering and probing for LLM safety](https://mlsanity.com/mechinterp/steering-and-probing-for-llm-safety/)

## Builds on (most cited first, 30 of 41)

- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [The Intrinsic Dimension of Images and Its Impact on Learning](https://mlsanity.com/paper/2104.08894/) (2021)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [Estimating the intrinsic dimension of datasets by a minimal neighborhood information](https://mlsanity.com/paper/1803.06992/) (2018)
- [ShieldGemma: Generative AI Content Moderation Based on Gemma](https://mlsanity.com/paper/2407.21772/) (2024)
- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [The emergence of clusters in self-attention dynamics](https://mlsanity.com/paper/2305.05465/) (2023)
- [Sinkformers: Transformers with Doubly Stochastic Attention](https://mlsanity.com/paper/2110.11773/) (2021)
- [The geometry of hidden representations of large transformer models](https://mlsanity.com/paper/2302.00294/) (2023)
- [Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts](https://mlsanity.com/paper/2306.04723/) (2023)
- [A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation](https://mlsanity.com/paper/2305.11391/) (2023)
- [Residual Connections Encourage Iterative Inference](https://mlsanity.com/paper/1710.04773/) (2017)
- [Testing the Predictions of Surprisal Theory in 11 Languages](https://mlsanity.com/paper/2307.03667/) (2023)
- [Scikit-dimension: a Python package for intrinsic dimension estimation](https://mlsanity.com/paper/2109.02596/) (2021)
- [Classification and Geometry of General Perceptual Manifolds](https://mlsanity.com/paper/1710.06487/) (2017)
- [Emergence of a High-Dimensional Abstraction Phase in Language Transformers](https://mlsanity.com/paper/2405.15471/) (2024)
- [A Mathematical Theory of Attention](https://mlsanity.com/paper/2007.02876/) (2020)
- [Dynamic metastability in the self-attention model](https://mlsanity.com/paper/2410.06833/) (2024)
- [How Smooth Is Attention?](https://mlsanity.com/paper/2312.14820/) (2023)
- [Does Representation Matter? Exploring Intermediate Layers in Large Language Models](https://mlsanity.com/paper/2412.09563/) (2024)
- [Hierarchical nucleation in deep neural networks](https://mlsanity.com/paper/2007.03506/) (2020)

## Built on by (most cited first, 17 of 17)

- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Beyond Exponential Decay: Rethinking Error Accumulation in Large Language Models](https://mlsanity.com/paper/2505.24187/) (2025)
- [Position: Foundation Models Need Digital Twin Representations](https://mlsanity.com/paper/2505.03798/) (2025)
- [On the Predictive Power of Representation Dispersion in Language Models](https://mlsanity.com/paper/2506.24106/) (2025)
- [From Topology to Retrieval: Decoding Embedding Spaces with Unified Signatures](https://mlsanity.com/paper/2511.22150/) (2025)
- [Latent Semantic Manifolds in Large Language Models](https://mlsanity.com/paper/2603.22301/) (2026)
- [Geometric Metrics and LLMs: What They Measure and When They Work](https://mlsanity.com/paper/2509.25359/) (2025)
- [Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlation](https://mlsanity.com/paper/2503.22547/) (2025)
- [IdEst: Assessing Self-Supervised Learning Representations via Intrinsic Dimension](https://mlsanity.com/paper/2606.03338/) (2026)
- [What's in a prompt? Language models encode literary style in prompt embeddings](https://mlsanity.com/paper/2505.17071/) (2025)
- [Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story](https://mlsanity.com/paper/2511.15210/) (2025)
- [MIRROR: Aligning Semantic Relations from Language to Image via Gromov--Wasserstein](https://mlsanity.com/paper/2606.29462/) (2026)
- [An Analysis of Residual-Stream Geometry Across Transformer Depth](https://mlsanity.com/paper/2607.18348/) (2026)
- [Probing Geometry of Next Token Prediction Using Cumulant Expansion of the Softmax Entropy](https://mlsanity.com/paper/2510.04285/) (2025)
- [Residual Stream Analysis of Overfitting And Structural Disruptions](https://mlsanity.com/paper/2603.13318/) (2026)
- [Rethinking Intrinsic Dimension Estimation in Neural Representations](https://mlsanity.com/paper/2604.20276/) (2026)
- [Isotropy Cliffs: The Geometric Signature of Decision-Making in Large Language Models](https://mlsanity.com/paper/2608.00828/) (2026)

## Abstract (arXiv)

We study the geometry of token representations at the prompt level in large language models through the lens of intrinsic dimension. Viewing transformers as mean-field particle systems, we estimate the intrinsic dimension of the empirical measure at each layer and demonstrate that it correlates with next-token uncertainty. Across models and intrinsic dimension estimators, we find that intrinsic dimension peaks in early to middle layers and increases under syntactic and semantic disruption (by shuffling tokens), and that it is strongly correlated with average surprisal, with a simple analysis linking logits geometry to entropy via softmax. As a case study in practical interpretability and safety, we train a linear probe on the per-layer intrinsic dimension profile to distinguish malicious from benign prompts before generation. This probe achieves accuracy of 90 to 95% in different datasets, outperforming widely used guardrails such as Llama Guard and Shield Gemma. We further compare against linear probes built from layerwise entropy derived via the Tuned Lens and find that the intrinsic dimension-based probe is competitive and complementary, offering a compact, interpretable signal distributed across layers. Our findings suggest that prompt-level geometry provides actionable signals for monitoring and controlling LLM behavior, and offers a bridge between mechanistic insights and practical safety tools.