mlsanityData to 7 Oct 2026

A Hyperbolicity Atlas of Large Language Model Hidden States

arXiv 2609.07053 ↗First posted 7 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

LLM hidden states are ordinary vectors, but the distances among those vectors may still show hierarchical structure. To our knowledge, this paper is the first systematic study of whether prompt-token hidden states in contemporary LLMs exhibit Gromov Hyperbolicity (GH), a distance-based measure of tree-likeness. Using 818,904 sample-layer measurements from ten open-weight models across MATH500, HumanEval, WinoGrande, and TruthfulQA, we build a GH map over four axes: parameter scale, layer depth, model family, and input domain. The clearest pattern is depth, not scale: middle layers usually form a high-relative-hyperbolicity plateau, while final layers often become substantially more tree-like. Scale effects are weak and non-monotonic, matched 7/8B model families differ strongly, and domains interact with model specialization. These findings make GH useful as a practical diagnostic: it shows where hierarchical distance structure appears, how specialization changes it, and which model-layer-domain comparisons deserve closer analysis.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 19 references that are in the map. First posted on arXiv 7 Sep 2026.

Builds on

14 of 19 references.

References · 14

  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  2. 2023Let's Verify Step by Stepcited by 3,598
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  5. 2017Poincar'e Embeddings for Learning Hierarchical Representationscited by 674
  6. 2018Hyperbolic Neural Networkscited by 393
  7. 2018Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometrycited by 261
  8. 2018Poincar'e GloVe: Hyperbolic Word Embeddingscited by 155
  9. 2023The geometry of hidden representations of large transformer modelscited by 108
  10. 2018Embedding Text in Hyperbolic Spacescited by 76
  11. 2025HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Expertscited by 22
  12. 2025The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilitiescited by 7
2 more
  1. 2025TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottleneckscited by 4
  2. 2026Lyapunov Probes for Hallucination Detection in Large Foundation Modelscited by 3

Built on by

Not cited yet.