# Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information

arXiv: https://arxiv.org/abs/2311.11509 · HTML: https://mlsanity.com/paper/2311.11509/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 12 of its 14 references that are in the map.
- First posted on arXiv 20 Nov 2023.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 12 of 12)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://mlsanity.com/paper/2309.00614/) (2023)
- [SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2310.03684/) (2023)
- [Detecting Language Model Attacks with Perplexity](https://mlsanity.com/paper/2308.14132/) (2023)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://mlsanity.com/paper/1605.07725/) (2016)
- [Certifying LLM Safety against Adversarial Prompting](https://mlsanity.com/paper/2309.02705/) (2023)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019)
- [Adversarial Training for Large Neural Language Models](https://mlsanity.com/paper/2004.08994/) (2020)

## Built on by (most cited first, 30 of 33)

- [Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey](https://mlsanity.com/paper/2402.09283/) (2024)
- [Know Your Limits: A Survey of Abstention in Large Language Models](https://mlsanity.com/paper/2407.18418/) (2024)
- [A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models](https://mlsanity.com/paper/2402.13457/) (2024)
- [Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses](https://mlsanity.com/paper/2406.01288/) (2024)
- [The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness](https://mlsanity.com/paper/2401.00287/) (2023)
- [Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents](https://mlsanity.com/paper/2411.09523/) (2024)
- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspective](https://mlsanity.com/paper/2401.06824/) (2024)
- [Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)](https://mlsanity.com/paper/2407.14937/) (2024)
- [Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models](https://mlsanity.com/paper/2403.04786/) (2024)
- [Securing Large Language Models: Threats, Vulnerabilities and Responsible Practices](https://mlsanity.com/paper/2403.12503/) (2024)
- [When "Competency" in Reasoning Opens the Door to Vulnerability: Jailbreaking LLMs via Novel Complex Ciphers](https://mlsanity.com/paper/2402.10601/) (2024)
- [PAPILLON: Efficient and Stealthy Fuzz Testing-Powered Jailbreaks for LLMs](https://mlsanity.com/paper/2409.14866/) (2024)
- [Emerging Vulnerabilities in Frontier Models: Multi-Turn Jailbreak Attacks](https://mlsanity.com/paper/2409.00137/) (2024)
- [Defense against Prompt Injection Attacks via Mixture of Encodings](https://mlsanity.com/paper/2504.07467/) (2025)
- [GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs](https://mlsanity.com/paper/2411.14133/) (2024)
- [Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerations](https://mlsanity.com/paper/2410.09097/) (2024)
- [OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalities](https://mlsanity.com/paper/2505.23856/) (2025)
- [LLM Safeguard is a Double-Edged Sword: Exploiting False Positives for Denial-of-Service Attacks](https://mlsanity.com/paper/2410.02916/) (2024)
- [Can LLMs be Fooled? Investigating Vulnerabilities in LLMs](https://mlsanity.com/paper/2407.20529/) (2024)
- [DeRAG: Black-box Adversarial Attacks on Multiple Retrieval-Augmented Generation Applications via Prompt Injection](https://mlsanity.com/paper/2507.15042/) (2025)
- [Data to Defense: The Role of Curation in Customizing LLMs Against Jailbreaking Attacks](https://mlsanity.com/paper/2410.02220/) (2024)
- [HateSieve: A Contrastive Learning Framework for Detecting and Segmenting Hateful Content in Multimodal Memes](https://mlsanity.com/paper/2408.05794/) (2024)
- [Geometry-Guided Adversarial Prompt Detection via Curvature and Local Intrinsic Dimension](https://mlsanity.com/paper/2503.03502/) (2025)
- [Disentangling Intent from Role: Adversarial Self-Play for Persona-Invariant Safety Alignment](https://mlsanity.com/paper/2605.01899/) (2026)
- [LaCache: Robust Semantic Caching for LLM Serving](https://mlsanity.com/paper/2608.01718/) (2026)
- [SafeBehavior: Simulating Human-Like Multistage Reasoning to Mitigate Jailbreak Attacks in Large Language Models](https://mlsanity.com/paper/2509.26345/) (2025)
- [CLASP: Defending Hybrid Large Language Models Against Hidden State Poisoning Attacks](https://mlsanity.com/paper/2603.12206/) (2026)
- [Is my model perplexed for the right reason? Contrasting LLMs' Benchmark Behavior with Token-Level Perplexity](https://mlsanity.com/paper/2603.29396/) (2026)
- [Detecting Fluent Optimization-Based Adversarial Prompts via Sequential Entropy Changes](https://mlsanity.com/paper/2605.19966/) (2026)

## Abstract (arXiv)

In recent years, Large Language Models (LLM) have emerged as pivotal tools in various applications. However, these models are susceptible to adversarial prompt attacks, where attackers can carefully curate input strings that mislead LLMs into generating incorrect or undesired outputs. Previous work has revealed that with relatively simple yet effective attacks based on discrete optimization, it is possible to generate adversarial prompts that bypass moderation and alignment of the models. This vulnerability to adversarial prompts underscores a significant concern regarding the robustness and reliability of LLMs. Our work aims to address this concern by introducing a novel approach to detecting adversarial prompts at a token level, leveraging the LLM's capability to predict the next token's probability. We measure the degree of the model's perplexity, where tokens predicted with high probability are considered normal, and those exhibiting high perplexity are flagged as adversarial. Additionaly, our method also integrates context understanding by incorporating neighboring token information to encourage the detection of contiguous adversarial prompt sequences. To this end, we design two algorithms for adversarial prompt detection: one based on optimization techniques and another on Probabilistic Graphical Models (PGM). Both methods are equipped with efficient solving methods, ensuring efficient adversarial prompt detection. Our token-level detection result can be visualized as heatmap overlays on the text sequence, allowing for a clearer and more intuitive representation of which part of the text may contain adversarial prompts.