# Geometric Metrics and LLMs: What They Measure and When They Work

arXiv: https://arxiv.org/abs/2509.25359 · HTML: https://mlsanity.com/paper/2509.25359/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 23 of 23)

- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank](https://mlsanity.com/paper/2210.02885/) (2022)
- [Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts](https://mlsanity.com/paper/2306.04723/) (2023)
- [On the Use of ArXiv as a Dataset](https://mlsanity.com/paper/1905.00075/) (2019)
- [Anisotropy Is Inherent to Self-Attention in Transformers](https://mlsanity.com/paper/2401.12143/) (2024)
- [Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension](https://mlsanity.com/paper/2402.18048/) (2024)
- [The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models](https://mlsanity.com/paper/2311.05928/) (2023)
- [A Survey on Large Language Model Benchmarks](https://mlsanity.com/paper/2508.15361/) (2025)
- [Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models](https://mlsanity.com/paper/2401.17139/) (2024)
- [LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures](https://mlsanity.com/paper/2312.04000/) (2023)
- [BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training](https://mlsanity.com/paper/2307.03131/) (2023)
- [Ground Every Sentence: Improving Retrieval-Augmented LLMs with Interleaved Reference-Claim Generation](https://mlsanity.com/paper/2407.01796/) (2024)
- [Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions](https://mlsanity.com/paper/2404.09135/) (2024)
- [The Intrinsic Dimension of Prompts in Internal Representations of Large Language Models](https://mlsanity.com/paper/2501.10573/) (2025)
- [Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Models](https://mlsanity.com/paper/2502.15799/) (2025)
- [Feature-Level Insights into Artificial Text Detection with Sparse Autoencoders](https://mlsanity.com/paper/2503.03601/) (2025)
- [Scaling Parameter-Constrained Language Models with Quality Data](https://mlsanity.com/paper/2410.03083/) (2024)
- [Automated Evaluation of Personalized Text Generation using Large Language Models](https://mlsanity.com/paper/2310.11593/) (2023)
- [Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMs](https://mlsanity.com/paper/2506.05387/) (2025)
- [Towards Dataset-scale and Feature-oriented Evaluation of Text Summarization in Large Language Model Prompts](https://mlsanity.com/paper/2407.12192/) (2024)

## Built on by (most cited first, 4 of 4)

- [Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Intervention](https://mlsanity.com/paper/2605.05892/) (2026)
- [Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story](https://mlsanity.com/paper/2511.15210/) (2025)
- [Bug or Feature2: Weight Drift, Activation Sparsity and Spikes](https://mlsanity.com/paper/2605.17659/) (2026)
- [Reasoning as State Transition: A Representational Analysis of Reasoning Evolution in Large Language Models](https://mlsanity.com/paper/2602.00770/) (2026)

## Abstract (arXiv)

We present a systematic stress-test of geometric metrics for LLM evaluation. Rank-based geometric properties of internal representations have shown promise as reference-free quality signals, but the conditions under which they are reliable remain unclear. We evaluate eight commonly-used metrics: intrinsic-dimensionality estimators, spectral norms, and related quantities across six tester models (0.5-8B) and eight generators on contrasting tasks, separating genuine geometric signal from text-length effects and from what standard text statistics already capture. Three findings emerge. First, some metrics (notably Schatten Norm and MOM) mainly reflect output length, and their apparent discriminative power collapses once length is controlled. Second, geometric metrics add modest but real information beyond text statistics: combined with them, a classifier reaches 78% accuracy on 6-way generator identification versus 69% for text statistics alone. Third, rather than tracking a general notion of text quality, the metrics demonstrate only moderate association between the intrinsic-dimensionality and lexical diversity (RTTR). We give use-case-specific recommendations and identify failure detection as the most promising near-term application.