# Token embeddings violate the manifold hypothesis

arXiv: https://arxiv.org/abs/2504.01002 · HTML: https://mlsanity.com/paper/2504.01002/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 70 references that are in the map.
- First posted on arXiv 1 Apr 2025.

## Builds on (most cited first, 30 of 34)

- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023)
- [Estimating the intrinsic dimension of datasets by a minimal neighborhood information](https://mlsanity.com/paper/1803.06992/) (2018)
- [Scaling Large Language Model-based Multi-Agent Collaboration](https://mlsanity.com/paper/2406.07155/) (2024)
- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [Does Prompt Formatting Have Any Impact on LLM Performance?](https://mlsanity.com/paper/2411.10541/) (2024)
- [Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View](https://mlsanity.com/paper/1906.02762/) (2019)
- [Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts](https://mlsanity.com/paper/2306.04723/) (2023)
- [Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration](https://mlsanity.com/paper/2404.12715/) (2024)
- [Hidden symmetries of ReLU networks](https://mlsanity.com/paper/2306.06179/) (2023)
- [Shared Global and Local Geometry of Language Model Embeddings](https://mlsanity.com/paper/2503.21073/) (2025)
- [LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law](https://mlsanity.com/paper/2402.00795/) (2024)
- [Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection](https://mlsanity.com/paper/2404.09894/) (2024)
- [Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations](https://mlsanity.com/paper/2408.15417/) (2024)
- [An enriched category theory of language: from syntax to semantics](https://mlsanity.com/paper/2106.07890/) (2021)
- [Limits for Learning with Language Models](https://mlsanity.com/paper/2306.12213/) (2023)
- [On the Brittle Foundations of ReAct Prompting for Agentic Large Language Models](https://mlsanity.com/paper/2405.13966/) (2024)
- [Topology of Word Embeddings: Singularities Reflect Polysemy](https://mlsanity.com/paper/2011.09413/) (2020)
- [The Unreasonable Effectiveness of Eccentric Automatic Prompts](https://mlsanity.com/paper/2402.10949/) (2024)
- [Topological Singularity Detection at Multiple Scales](https://mlsanity.com/paper/2210.00069/) (2022)
- [The structure of the token space for large language models](https://mlsanity.com/paper/2410.08993/) (2024)
- [Transparency Helps Reveal When Language Models Learn Meaning](https://mlsanity.com/paper/2210.07468/) (2022)
- [Estimating the reach of a manifold via its convexity defect function](https://mlsanity.com/paper/2001.08006/) (2020)
- [The Information of Large Language Model Geometry](https://mlsanity.com/paper/2402.03471/) (2024)
- [Building Intelligence Identification System via Large Language Model Watermarking: A Survey and Beyond](https://mlsanity.com/paper/2407.11100/) (2024)
- [The magnitude of categories of texts enriched by language models](https://mlsanity.com/paper/2501.06662/) (2025)
- [Measuring Intrinsic Dimension of Token Embeddings](https://mlsanity.com/paper/2503.02142/) (2025)

## Built on by (most cited first, 16 of 16)

- [Beyond Exponential Decay: Rethinking Error Accumulation in Large Language Models](https://mlsanity.com/paper/2505.24187/) (2025)
- [The Information Geometry of Softmax: Probing and Steering](https://mlsanity.com/paper/2602.15293/) (2026)
- [Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMs](https://mlsanity.com/paper/2511.07419/) (2025)
- [Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA](https://mlsanity.com/paper/2602.22617/) (2026)
- [Latent Semantic Manifolds in Large Language Models](https://mlsanity.com/paper/2603.22301/) (2026)
- [Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics](https://mlsanity.com/paper/2604.08764/) (2026)
- [A Deep Generative Approach to Stratified Learning](https://mlsanity.com/paper/2604.10650/) (2026)
- [Exploring the Stratified Space Structure of an RL Game with the Volume Growth Transform](https://mlsanity.com/paper/2507.22010/) (2025)
- [Advancing mathematics research with generative AI](https://mlsanity.com/paper/2511.07420/) (2025)
- [Decoding in Geometry: Alleviating Embedding-Space Crowding for Complex Reasoning](https://mlsanity.com/paper/2601.22536/) (2026)
- [Stratifying Reinforcement Learning with Signal Temporal Logic](https://mlsanity.com/paper/2604.04923/) (2026)
- [Rethinking Intrinsic Dimension Estimation in Neural Representations](https://mlsanity.com/paper/2604.20276/) (2026)
- [Manifold Bandits: Bayesian Curriculum Learning over the Latent Geometry of Large Language Models](https://mlsanity.com/paper/2606.19750/) (2026)
- [Riemannian Geometry for Pre-trained Language Model Embeddings](https://mlsanity.com/paper/2607.07047/) (2026)
- [Geometric Filtering of LLM-Generated Samples for Few-Shot Text Classification](https://mlsanity.com/paper/2608.13866/) (2026)
- [A Hub of Short Rows Inflates Intrinsic Dimension Estimation of Token Embeddings](https://mlsanity.com/paper/2608.29702/) (2026)

## Abstract (arXiv)

A full understanding of the behavior of a large language model (LLM) requires our grasp of its input token space. If this space differs from our assumptions, our comprehension of and conclusions about the LLM will likely be flawed. We elucidate the structure of the token embeddings both empirically and theoretically. We present a novel statistical test assuming that the neighborhood around each token has a relatively flat and smooth structure as the null hypothesis. Failing to reject the null is uninformative, but rejecting it at a specific token psi implies an irregularity in the token subspace in a psi-neighborhood, B(psi). The structure assumed in the null is a generalization of a manifold with boundary called a smooth fiber bundle (which can be split into two spatial regimes -- small and large radius), so we denote our new hypothesis test as the ``fiber bundle hypothesis.'' By running our test over several open-source LLMs, each with unique token embeddings, we find that the null is frequently rejected, and so the evidence suggests that the token subspace is not a fiber bundle and hence also not a manifold. As a consequence of our findings, when an LLM is presented with two semantically equivalent prompts, if one prompt contains a token implicated by our test, the response to that prompt will likely exhibit less stability than the other.