mlsanityData to 7 Oct 2026

Single-pass Detection of Jailbreaking Input in Large Language Models

arXiv 2502.15435 ↗First posted 21 Feb 2025cs.LG cs.AI cs.CLcited by 10

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Defending aligned Large Language Models (LLMs) against jailbreaking attacks is a challenging problem, with existing approaches requiring multiple requests or even queries to auxiliary LLMs, making them computationally heavy. Instead, we focus on detecting jailbreaking input in a single forward pass. Our method, called Single Pass Detection SPD, leverages the information carried by the logits to predict whether the output sentence will be harmful. This allows us to defend in just one forward pass. SPD can not only detect attacks effectively on open-source models, but also minimizes the misclassification of harmless inputs. Furthermore, we show that SPD remains effective even without complete logit access in GPT-3.5 and GPT-4. We believe that our proposed method offers a promising approach to efficiently safeguard LLMs against adversarial attacks.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 72 of its 92 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

72 of 92 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey26 Sep
  2. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  3. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models27 May
  4. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing25 Feb
  5. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  7. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  8. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  9. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  11. 2023Are aligned neural networks adversarially aligned?26 Jun

Other references · 61

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2013Intriguing properties of neural networkscited by 5,114
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  10. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
38 more
  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  2. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  4. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  5. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  6. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  7. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  8. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  9. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  10. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  11. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  12. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  13. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  14. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  15. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  16. 2018Generating Natural Language Adversarial Examplescited by 422
  17. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  18. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  19. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  20. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  21. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
  22. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  23. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  24. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  25. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  26. 2015Inverting Visual Representations with Convolutional Networkscited by 168
  27. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  28. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  29. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  30. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  31. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  32. 2023An LLM can Fool Itself: A Prompt-Based Adversarial Attackcited by 111
  33. 2023Regulating ChatGPT and other Large Generative AI Modelscited by 102
  34. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  35. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  36. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  37. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 83
  38. 2021Improving Question Answering Model Robustness with Synthetic Adversarial Data Generationcited by 77

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2025LLM Jailbreak Detection for (Almost) Free!cited by 13
  3. 2025Think Twice, Generate Once: Safeguarding by Progressive Self-Reflectioncited by 11
  4. 2025Machine Unlearning Meets Adversarial Robustness via Constrained Interventions on LLMscited by 3
  5. 2026Jailbreaking Leaves a Trace: Understanding and Detecting Jailbreak Attacks from Internal Representations of Large Language Modelscited by 3
  6. 2026AISA: Awakening Intrinsic Safety Awareness in Large Language Models against Jailbreak Attackscited by 3
  7. 2025Jailbreaking in the Haystackcited by 2
  8. 2026What Intermediate Layers Know: Detecting Jailbreaks from Entropy Dynamicscited by 1
  9. 2026Detecting Fluent Optimization-Based Adversarial Prompts via Sequential Entropy Changescited by 0
  10. 2026Beyond Attack Success Rate: Temporal Logit Observability for LLM Safety Failurescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024textitMMJ-Bench: A Comprehensive Study on Jailbreak Attacks and Defenses for Multimodal Large Language Models
  2. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks
  3. 2024PrisonBreak: Jailbreaking Large Language Models with at Most Twenty-Five Targeted Bit-flips
  4. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  5. 2025The Tower of Babel Revisited: Multilingual Jailbreak Prompts on Closed-Source Large Language Models
  6. 2024Defending against Jailbreak through Early Exit Generation of Large Language Models
  7. 2026Defending Large Language Models Against Jailbreak Attacks via In-Decoding Safety-Awareness Probing
  8. 2025Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers