# Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection

arXiv: https://arxiv.org/abs/2404.09894 · HTML: https://mlsanity.com/paper/2404.09894/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 52 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023)
- [From Louvain to Leiden: guaranteeing well-connected communities](https://mlsanity.com/paper/1810.08473/) (2018)
- [Text and Code Embeddings by Contrastive Pre-Training](https://mlsanity.com/paper/2201.10005/) (2022)
- [MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots](https://mlsanity.com/paper/2307.08715/) (2023)
- [PentestGPT: An LLM-empowered Automatic Penetration Testing Tool](https://mlsanity.com/paper/2308.06782/) (2023)
- [CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?](https://mlsanity.com/paper/2306.16636/) (2023)
- [Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues](https://mlsanity.com/paper/2402.09091/) (2024)
- [Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoning](https://mlsanity.com/paper/2402.08416/) (2024)
- [BiasAsker: Measuring the Bias in Conversational AI System](https://mlsanity.com/paper/2305.12434/) (2023)
- [Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPT](https://mlsanity.com/paper/2304.02014/) (2023)
- [NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers](https://mlsanity.com/paper/2207.13066/) (2022)
- [MTTM: Metamorphic Testing for Textual Content Moderation Software](https://mlsanity.com/paper/2302.05706/) (2023)
- [BARD: A structured technique for group elicitation of Bayesian networks to support analytic reasoning](https://mlsanity.com/paper/2003.01207/) (2020)
- [E-NER: Evidential Deep Learning for Trustworthy Named Entity Recognition](https://mlsanity.com/paper/2305.17854/) (2023)
- [Validating Multimedia Content Moderation Software via Semantic Fusion](https://mlsanity.com/paper/2305.13623/) (2023)
- [MeTMaP: Metamorphic Testing for Detecting False Vector Matching Problems in LLM Augmented Generation](https://mlsanity.com/paper/2402.14480/) (2024)
- [Automated Testing and Improvement of Named Entity Recognition Systems](https://mlsanity.com/paper/2308.07937/) (2023)
- [An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software](https://mlsanity.com/paper/2308.09810/) (2023)

## Built on by (most cited first, 29 of 29)

- [LLMmap: Fingerprinting For Large Language Models](https://mlsanity.com/paper/2407.15847/) (2024)
- [garak: A Framework for Security Probing Large Language Models](https://mlsanity.com/paper/2406.11036/) (2024)
- [BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training](https://mlsanity.com/paper/2409.04599/) (2024)
- [Token embeddings violate the manifold hypothesis](https://mlsanity.com/paper/2504.01002/) (2025)
- [Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysis](https://mlsanity.com/paper/2410.16527/) (2024)
- [From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agents](https://mlsanity.com/paper/2505.12981/) (2025)
- [Understanding the Effectiveness of Coverage Criteria for Large Language Models: A Special Angle from Jailbreak Attacks](https://mlsanity.com/paper/2408.15207/) (2024)
- [Constructing a BPE Tokenization DFA](https://mlsanity.com/paper/2405.07671/) (2024)
- [Repairing Catastrophic-Neglect in Text-to-Image Diffusion Models via Attention-Guided Feature Enhancement](https://mlsanity.com/paper/2406.16272/) (2024)
- [GlitchProber: Advancing Effective Detection and Mitigation of Glitch Tokens in Large Language Models](https://mlsanity.com/paper/2408.04905/) (2024)
- [Speculating LLMs' Chinese Training Data Pollution from Their Tokens](https://mlsanity.com/paper/2508.17771/) (2025)
- [GlitchMiner: Mining Glitch Tokens in Large Language Models via Gradient-based Discrete Optimization](https://mlsanity.com/paper/2410.15052/) (2024)
- [Addressing Tokenization Inconsistency in Steganography and Watermarking Based on Large Language Models](https://mlsanity.com/paper/2508.20718/) (2025)
- [Breaking the Loop: Detecting and Mitigating Denial-of-Service Vulnerabilities in Large Language Models](https://mlsanity.com/paper/2503.00416/) (2025)
- [Probing the topology of the space of tokens with structured prompts](https://mlsanity.com/paper/2503.15421/) (2025)
- [AdaptBPE: From General Purpose to Specialized Tokenizers](https://mlsanity.com/paper/2601.21665/) (2026)
- [LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers](https://mlsanity.com/paper/2602.04706/) (2026)
- [Disentangling MLP Neuron Weights in Vocabulary Space](https://mlsanity.com/paper/2604.06005/) (2026)
- [Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuation](https://mlsanity.com/paper/2507.08020/) (2025)
- [Can Large Language Models put 2 and 2 together? Probing for Entailed Arithmetical Relationships](https://mlsanity.com/paper/2404.19432/) (2024)
- [Demystifying optimized prompts in language models](https://mlsanity.com/paper/2505.02273/) (2025)
- [Trusted Weights, Treacherous Optimizations? Optimization-Triggered Backdoor Attacks on LLMs](https://mlsanity.com/paper/2605.20641/) (2026)
- [Logit-Gap Steering: A Forward-Pass Diagnostic for Alignment Robustness](https://mlsanity.com/paper/2506.24056/) (2025)
- [TokenBlowUp: Resolving Representational Singularities in LLM Token Spaces via Monoidal Transformations](https://mlsanity.com/paper/2507.19747/) (2025)
- [Generative Chemical Language Models for Energetic Materials Discovery](https://mlsanity.com/paper/2604.03304/) (2026)
- [Check Your LLM's Secret Dictionary! Five Lines of Code Reveal What Your LLM Learned (Including What It Shouldn't Have)](https://mlsanity.com/paper/2605.22005/) (2026)
- [Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models](https://mlsanity.com/paper/2606.14122/) (2026)
- [Out of Sight: Compression-Aware Content Protection against Agentic Crawlers](https://mlsanity.com/paper/2607.08180/) (2026)
- [GlitchPatch: Repairing Glitch Tokens in Frozen Language Models via Local Retokenization](https://mlsanity.com/paper/2610.04399/) (2026)

## Abstract (arXiv)

With the expanding application of Large Language Models (LLMs) in various domains, it becomes imperative to comprehensively investigate their unforeseen behaviors and consequent outcomes. In this study, we introduce and systematically explore the phenomenon of "glitch tokens", which are anomalous tokens produced by established tokenizers and could potentially compromise the models' quality of response. Specifically, we experiment on seven top popular LLMs utilizing three distinct tokenizers and involving a totally of 182,517 tokens. We present categorizations of the identified glitch tokens and symptoms exhibited by LLMs when interacting with glitch tokens. Based on our observation that glitch tokens tend to cluster in the embedding space, we propose GlitchHunter, a novel iterative clustering-based technique, for efficient glitch token detection. The evaluation shows that our approach notably outperforms three baseline methods on eight open-source LLMs. To the best of our knowledge, we present the first comprehensive study on glitch tokens. Our new detection further provides valuable insights into mitigating tokenization-related errors in LLMs.