# Probing the topology of the space of tokens with structured prompts

arXiv: https://arxiv.org/abs/2503.15421 · HTML: https://mlsanity.com/paper/2503.15421/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 19 Mar 2025.

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Llemma: An Open Language Model For Mathematics](https://mlsanity.com/paper/2310.10631/) (2023)
- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts](https://mlsanity.com/paper/2306.04723/) (2023)
- [Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models](https://mlsanity.com/paper/2405.05417/) (2024)
- [Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection](https://mlsanity.com/paper/2404.09894/) (2024)
- [An enriched category theory of language: from syntax to semantics](https://mlsanity.com/paper/2106.07890/) (2021)
- [Topology of Word Embeddings: Singularities Reflect Polysemy](https://mlsanity.com/paper/2011.09413/) (2020)
- [The structure of the token space for large language models](https://mlsanity.com/paper/2410.08993/) (2024)
- [GlitchProber: Advancing Effective Detection and Mitigation of Glitch Tokens in Large Language Models](https://mlsanity.com/paper/2408.04905/) (2024)
- [The magnitude of categories of texts enriched by language models](https://mlsanity.com/paper/2501.06662/) (2025)
- [A Language and Its Dimensions: Intrinsic Dimensions of Language Fractal Structures](https://mlsanity.com/paper/2311.10217/) (2023)
- [Twisty Takens: A Geometric Characterization of Good Observations on Dense Trajectories](https://mlsanity.com/paper/1809.07131/) (2018)

## Built on by (most cited first, 3 of 3)

- [Token embeddings violate the manifold hypothesis](https://mlsanity.com/paper/2504.01002/) (2025)
- [Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics](https://mlsanity.com/paper/2604.08764/) (2026)
- [TokenBlowUp: Resolving Representational Singularities in LLM Token Spaces via Monoidal Transformations](https://mlsanity.com/paper/2507.19747/) (2025)

## Abstract (arXiv)

This article presents a general and flexible method for prompting a large language model (LLM) to reveal its (hidden) token input embedding up to homeomorphism. Moreover, this article provides strong theoretical justification -- a mathematical proof for generic LLMs -- for why this method should be expected to work. With this method in hand, we demonstrate its effectiveness by recovering the token subspace of Llemma-7B. The results of this paper apply not only to LLMs but also to general nonlinear autoregressive processes.