mlsanityData to 14 Oct 2026

Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information

arXiv 2311.11509 ↗First posted 20 Nov 2023cs.CL cs.LGcited by 33

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

In recent years, Large Language Models (LLM) have emerged as pivotal tools in various applications. However, these models are susceptible to adversarial prompt attacks, where attackers can carefully curate input strings that mislead LLMs into generating incorrect or undesired outputs. Previous work has revealed that with relatively simple yet effective attacks based on discrete optimization, it is possible to generate adversarial prompts that bypass moderation and alignment of the models. This vulnerability to adversarial prompts underscores a significant concern regarding the robustness and reliability of LLMs. Our work aims to address this concern by introducing a novel approach to detecting adversarial prompts at a token level, leveraging the LLM's capability to predict the next token's probability. We measure the degree of the model's perplexity, where tokens predicted with high probability are considered normal, and those exhibiting high perplexity are flagged as adversarial. Additionaly, our method also integrates context understanding by incorporating neighboring token information to encourage the detection of contiguous adversarial prompt sequences. To this end, we design two algorithms for adversarial prompt detection: one based on optimization techniques and another on Probabilistic Graphical Models (PGM). Both methods are equipped with efficient solving methods, ensuring efficient adversarial prompt detection. Our token-level detection result can be visualized as heatmap overlays on the text sequence, allowing for a clearer and more intuitive representation of which part of the text may contain adversarial prompts.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 12 of its 14 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

12 of 14 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  2. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  3. 2023Detecting Language Model Attacks with Perplexity27 Aug
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 8

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  6. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  7. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 195
  8. 2020Adversarial Training for Large Neural Language Modelscited by 111

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 19; 2025: 5; 2026: 8.

33 papers.

Sort:

By area:

  1. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  2. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  3. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 106
  4. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 70
  5. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 67
  6. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  7. 2024Large Language Model Safety: A Holistic Surveycited by 59
  8. 2024Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspectivecited by 48
  9. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  10. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Modelscited by 44
  11. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 37
  12. 2024When "Competency" in Reasoning Opens the Door to Vulnerability: Jailbreaking LLMs via Novel Complex Cipherscited by 30
21 more
  1. 2024PAPILLON: Efficient and Stealthy Fuzz Testing-Powered Jailbreaks for LLMscited by 21
  2. 2024Emerging Vulnerabilities in Frontier Models: Multi-Turn Jailbreak Attackscited by 16
  3. 2025Defense against Prompt Injection Attacks via Mixture of Encodingscited by 14
  4. 2024GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMscited by 12
  5. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  6. 2025OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalitiescited by 7
  7. 2024LLM Safeguard is a Double-Edged Sword: Exploiting False Positives for Denial-of-Service Attackscited by 6
  8. 2024Can LLMs be Fooled? Investigating Vulnerabilities in LLMscited by 5
  9. 2025DeRAG: Black-box Adversarial Attacks on Multiple Retrieval-Augmented Generation Applications via Prompt Injectioncited by 5
  10. 2024Data to Defense: The Role of Curation in Customizing LLMs Against Jailbreaking Attackscited by 3
  11. 2024HateSieve: A Contrastive Learning Framework for Detecting and Segmenting Hateful Content in Multimodal Memescited by 2
  12. 2025Geometry-Guided Adversarial Prompt Detection via Curvature and Local Intrinsic Dimensioncited by 1
  13. 2026Disentangling Intent from Role: Adversarial Self-Play for Persona-Invariant Safety Alignmentcited by 1
  14. 2026LaCache: Robust Semantic Caching for LLM Servingcited by 1
  15. 2025SafeBehavior: Simulating Human-Like Multistage Reasoning to Mitigate Jailbreak Attacks in Large Language Modelscited by 0
  16. 2026CLASP: Defending Hybrid Large Language Models Against Hidden State Poisoning Attackscited by 0
  17. 2026Is my model perplexed for the right reason? Contrasting LLMs' Benchmark Behavior with Token-Level Perplexitycited by 0
  18. 2026Detecting Fluent Optimization-Based Adversarial Prompts via Sequential Entropy Changescited by 0
  19. 2026LCO: LLM-based Constraint Optimization for Safer Agentic LLMs in Real-world Taskscited by 0
  20. 2026Greedy Coordinate Diffusion: Effective and Semantically Coherent Adversarial Attacks via Diffusion Guidancecited by 0
  21. 2026DataRx: Missingness-Aware Sampling for Safer Large Language Model Task-Specific Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
  2. 2025Adversarial Tokenization
  3. 2026Disentangling Adversarial Prompts: A Semantic-Graph Defense for Robust LLM Security
  4. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models
  5. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMs
  6. 2025Safeguarding Efficacy in Large Language Models: Evaluating Resistance to Human-Written and Algorithmic Adversarial Prompts
  7. 2025CoCoTen: Detecting Adversarial Inputs to Large Language Models through Latent Space Features of Contextual Co-occurrence Tensors
  8. 2024Human-Interpretable Adversarial Prompt Attack on Large Language Models with Situational Context