# GlitchProber: Advancing Effective Detection and Mitigation of Glitch Tokens in Large Language Models

arXiv: https://arxiv.org/abs/2408.04905 · HTML: https://mlsanity.com/paper/2408.04905/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 40 references that are in the map.
- First posted on arXiv 9 Aug 2024.

## Builds on (most cited first, 14 of 14)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [Pay Attention to MLPs](https://mlsanity.com/paper/2105.08050/) (2021)
- [MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots](https://mlsanity.com/paper/2307.08715/) (2023)
- [Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoning](https://mlsanity.com/paper/2402.08416/) (2024)
- [Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization](https://mlsanity.com/paper/2405.17067/) (2024)
- [Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection](https://mlsanity.com/paper/2404.09894/) (2024)
- [Digger: Detecting Copyright Content Mis-usage in Large Language Model Training](https://mlsanity.com/paper/2401.00676/) (2024)
- [Simplified Self-Attention for Transformer-based End-to-End Speech Recognition](https://mlsanity.com/paper/2005.10463/) (2020)
- [Uncovering Logit Suppression Vulnerabilities in LLM Safety Alignment](https://mlsanity.com/paper/2405.13068/) (2024)
- [MeTMaP: Metamorphic Testing for Detecting False Vector Matching Problems in LLM Augmented Generation](https://mlsanity.com/paper/2402.14480/) (2024)

## Built on by (most cited first, 5 of 5)

- [Understanding the Effectiveness of Coverage Criteria for Large Language Models: A Special Angle from Jailbreak Attacks](https://mlsanity.com/paper/2408.15207/) (2024)
- [GlitchMiner: Mining Glitch Tokens in Large Language Models via Gradient-based Discrete Optimization](https://mlsanity.com/paper/2410.15052/) (2024)
- [Probing the topology of the space of tokens with structured prompts](https://mlsanity.com/paper/2503.15421/) (2025)
- [Check Your LLM's Secret Dictionary! Five Lines of Code Reveal What Your LLM Learned (Including What It Shouldn't Have)](https://mlsanity.com/paper/2605.22005/) (2026)
- [GlitchPatch: Repairing Glitch Tokens in Frozen Language Models via Local Retokenization](https://mlsanity.com/paper/2610.04399/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have achieved unprecedented success in the field of natural language processing. However, the black-box nature of their internal mechanisms has brought many concerns about their trustworthiness and interpretability. Recent research has discovered a class of abnormal tokens in the model's vocabulary space and named them "glitch tokens". Those tokens, once included in the input, may induce the model to produce incorrect, irrelevant, or even harmful results, drastically undermining the reliability and practicality of LLMs. In this work, we aim to enhance the understanding of glitch tokens and propose techniques for their detection and mitigation. We first reveal the characteristic features induced by glitch tokens on LLMs, which are evidenced by significant deviations in the distributions of attention patterns and dynamic information from intermediate model layers. Based on the insights, we develop GlitchProber, a tool for efficient glitch token detection and mitigation. GlitchProber utilizes small-scale sampling, principal component analysis for accelerated feature extraction, and a simple classifier for efficient vocabulary screening. Taking one step further, GlitchProber rectifies abnormal model intermediate layer values to mitigate the destructive effects of glitch tokens. Evaluated on five mainstream open-source LLMs, GlitchProber demonstrates higher efficiency, precision, and recall compared to existing approaches, with an average F1 score of 0.86 and an average repair rate of 50.06%. GlitchProber unveils a novel path to address the challenges posed by glitch tokens and inspires future research toward more robust and interpretable LLMs.