mlsanityData to 7 Oct 2026

Detecting Language Model Attacks with Perplexity

arXiv 2308.14132 ↗First posted 27 Aug 2023cs.CL cs.AI cs.CRcited by 411

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

A novel hack involving Large Language Models (LLMs) has emerged, exploiting adversarial suffixes to deceive models into generating perilous responses. Such jailbreaks can trick LLMs into providing intricate instructions to a malicious user for creating explosives, orchestrating a bank heist, or facilitating the creation of offensive content. By evaluating the perplexity of queries with adversarial suffixes using an open-source LLM (GPT-2), we found that they have exceedingly high perplexity values. As we explored a broad range of regular (non-adversarial) prompt varieties, we concluded that false positives are a significant challenge for plain perplexity filtering. A Light-GBM trained on perplexity and token length resolved the false positives and correctly detected most adversarial attacks in the test set.

Key facts. Cited by 411 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 21 of its 34 references that are in the map. First posted on arXiv 27 Aug 2023.

Builds on

21 of 34 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  10. 2023Large Language Modelscited by 774
  11. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  12. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 439
7 more
  1. 2019DocRED: A Large-Scale Document-Level Relation Extraction Datasetcited by 239
  2. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 211
  3. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  4. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  5. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  6. 2021Globally-Robust Neural Networkscited by 106
  7. 2020Adversarial Attack and Defense of Structured Prediction Modelscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 115; 2025: 175; 2026: 106.

411 papers.

Sort:

By area:

  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  2. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  3. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  4. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  5. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
  6. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  7. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  8. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  9. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  10. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  11. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  12. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
38 more
  1. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  2. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  3. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  4. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  5. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  6. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  7. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  8. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  9. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  10. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  11. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 112
  12. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  13. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 102
  14. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 93
  15. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 90
  16. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  17. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  18. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 87
  19. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 82
  20. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  21. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 80
  22. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 77
  23. 2024Safeguarding Large Language Models: A Surveycited by 73
  24. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  25. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothingcited by 68
  26. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  27. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  28. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 67
  29. 2024Defending LLMs against Jailbreaking Attacks via Backtranslationcited by 65
  30. 2024Intention Analysis Makes LLMs A Good Jailbreak Defendercited by 64
  31. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63
  32. 2024Fight Back Against Jailbreaking via Prompt Adversarial Tuningcited by 63
  33. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  34. 2025A-MemGuard: A Proactive Defense Framework for LLM-Based Agent Memorycited by 61
  35. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  36. 2025Contextual Agent Security: A Policy for Every Purposecited by 55
  37. 2025Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Modelscited by 51
  38. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Railscited by 49

50 most cited of 411 All 411 (JSON)