# TokenBlowUp: Resolving Representational Singularities in LLM Token Spaces via Monoidal Transformations

arXiv: https://arxiv.org/abs/2507.19747 · HTML: https://mlsanity.com/paper/2507.19747/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 26 Jul 2025.

## Builds on (most cited first, 11 of 11)

- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View](https://mlsanity.com/paper/1906.02762/) (2019)
- [Manifold learning: what, how, and why](https://mlsanity.com/paper/2311.03757/) (2023)
- [Pure and Spurious Critical Points: a Geometric Study of Linear Networks](https://mlsanity.com/paper/1910.01671/) (2019)
- [LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law](https://mlsanity.com/paper/2402.00795/) (2024)
- [Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection](https://mlsanity.com/paper/2404.09894/) (2024)
- [Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoning](https://mlsanity.com/paper/2504.09772/) (2025)
- [Quantitative Clustering in Mean-Field Transformer Models](https://mlsanity.com/paper/2504.14697/) (2025)
- [The Information of Large Language Model Geometry](https://mlsanity.com/paper/2402.03471/) (2024)
- [The magnitude of categories of texts enriched by language models](https://mlsanity.com/paper/2501.06662/) (2025)
- [Probing the topology of the space of tokens with structured prompts](https://mlsanity.com/paper/2503.15421/) (2025)

## Abstract (arXiv)

Recent work has provided compelling evidence challenging the foundational manifold hypothesis for the token embedding spaces of Large Language Models (LLMs). These findings reveal the presence of geometric singularities around polysemous tokens, which can lead to representational instability. Existing methodologies, which presuppose a smooth data manifold, are ill-equipped to address such intrinsic structural flaws. In this paper, we formalize this problem in the language of scheme theory and propose a rigorous resolution by applying the scheme-theoretic blow-up at each singular point. This procedure replaces a singular point in the ambient affine scheme with its exceptional divisor, which we identify as a canonical geometric space -- a projective space of directions -- that houses the disambiguated semantic meanings of the token. This process of ``representational desingularization'' constructs a new geometric landscape for embeddings. We prove a formal theorem guaranteeing the geometric regularization of this new space, showing that the original pathologies are resolved. Finally, we outline the architectural implications of our framework, arguing for a paradigm shift from static look-ups to dynamic, geometrically-grounded computation.