mlsanityData to 7 Oct 2026

Probing the topology of the space of tokens with structured prompts

arXiv 2503.15421 ↗First posted 19 Mar 2025math.DG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

This article presents a general and flexible method for prompting a large language model (LLM) to reveal its (hidden) token input embedding up to homeomorphism. Moreover, this article provides strong theoretical justification -- a mathematical proof for generic LLMs -- for why this method should be expected to work. With this method in hand, we demonstrate its effectiveness by recovering the token subspace of Llemma-7B. The results of this paper apply not only to LLMs but also to general nonlinear autoregressive processes.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 16 of its 32 references that are in the map. First posted on arXiv 19 Mar 2025.

Builds on

16 of 32 references.

References · 16

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Mistral 7Bcited by 3,065
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2023Llemma: An Open Language Model For Mathematicscited by 385
  6. 2023A mathematical perspective on Transformerscited by 160
  7. 2023Intrinsic Dimension Estimation for Robust Detection of AI-Generated Textscited by 111
  8. 2024Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Modelscited by 45
  9. 2024Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detectioncited by 27
  10. 2021An enriched category theory of language: from syntax to semanticscited by 21
  11. 2020Topology of Word Embeddings: Singularities Reflect Polysemycited by 15
  12. 2024The structure of the token space for large language modelscited by 10
4 more
  1. 2025The magnitude of categories of texts enriched by language modelscited by 5
  2. 2023A Language and Its Dimensions: Intrinsic Dimensions of Language Fractal Structurescited by 4
  3. 2024GlitchProber: Advancing Effective Detection and Mitigation of Glitch Tokens in Large Language Modelscited by 3
  4. 2018Twisty Takens: A Geometric Characterization of Good Observations on Dense Trajectoriescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Token embeddings violate the manifold hypothesiscited by 14
  2. 2026Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamicscited by 3
  3. 2025TokenBlowUp: Resolving Representational Singularities in LLM Token Spaces via Monoidal Transformationscited by 0