mlsanityData to 14 Oct 2026

LLM Jailbreak Detection for (Almost) Free!

arXiv 2509.14558 ↗First posted 18 Sep 2025cs.CR cs.AI cs.CLcited by 13

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large language models (LLMs) enhance security through alignment when widely used, but remain susceptible to jailbreak attacks capable of producing inappropriate content. Jailbreak detection methods show promise in mitigating jailbreak attacks through the assistance of other models or multiple model inferences. However, existing methods entail significant computational costs. In this paper, we first present a finding that the difference in output distributions between jailbreak and benign prompts can be employed for detecting jailbreak prompts. Based on this finding, we propose a Free Jailbreak Detection (FJD) which prepends an affirmative instruction to the input and scales the logits by temperature to further distinguish between jailbreak and benign prompts through the confidence of the first token. Furthermore, we enhance the detection performance of FJD through the integration of virtual instruction learning. Extensive experiments on aligned LLMs show that our FJD can effectively detect jailbreak prompts with almost no additional computational costs during LLM inference.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 51 of its 61 references that are in the map. First posted on arXiv 18 Sep 2025.

Builds on

51 of 61 references.

Closest ancestors · 27

Same direction, newest first

  1. 2025Single-pass Detection of Jailbreaking Input in Large Language Models21 Feb
  2. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Models31 May
  3. 2024Boosting Jailbreak Attack with Momentum2 May
  4. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs21 Apr
  5. 2024Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?4 Apr
  6. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content19 Mar
  7. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks2 Mar
  8. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers25 Feb
  9. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis21 Feb
  10. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding14 Feb
  11. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization15 Nov
  12. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily14 Nov
15 more
  1. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  2. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  4. 2023Multilingual Jailbreak Challenges in Large Language Models10 Oct
  5. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  6. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  7. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  8. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  9. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  10. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  11. 2023Detecting Language Model Attacks with Perplexity27 Aug
  12. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  13. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  14. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  15. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 24

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  11. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  12. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
12 more
  1. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  2. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  3. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  4. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
  5. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  6. 2023Are aligned neural networks adversarially aligned?cited by 315
  7. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  8. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  9. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 191
  10. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  11. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  12. 2024Gradient-Based Language Model Red Teamingcited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 10.

13 papers.

Sort:

By area:

  1. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 11
  2. 2026Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMscited by 4
  3. 2026ALERT: Zero-shot LLM Jailbreak Detection via Internal Discrepancy Amplificationcited by 3
  4. 2025Do You Really Need a GPU to Guard Your LLM? CPU-Class Classifiers and Multi-Stage Pipelines for Safety Enforcement at Scalecited by 2
  5. 2026An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Modelscited by 2
  6. 2026RouteScan: A Non-Intrusive Approach to Auditing MoE LLMs Safety via Expert Routing Telemetrycited by 2
  7. 2025High-Entropy Tokens as Multimodal Failure Points in Vision-Language Modelscited by 1
  8. 2026What Intermediate Layers Know: Detecting Jailbreaks from Entropy Dynamicscited by 1
  9. 2026Efficient and Adaptable Detection of Malicious LLM Prompts via Bootstrap Aggregationcited by 0
  10. 2026Beyond Attack Success Rate: Temporal Logit Observability for LLM Safety Failurescited by 0
  11. 2026PRISM: Recovering Instruction Sets from Language Model Activationscited by 0
  12. 2026ForeSight: Enhancing Risk Monitoring via Early Safety Signal Distillationcited by 0
1 more
  1. 2026Why Jailbreaks Succeed in Diffusion Language Models: An Energy Landscape Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Re-Triggering Safeguards within LLMs for Jailbreak Detection
  2. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks
  3. 2025Machine Learning for Detection and Analysis of Novel LLM Jailbreaks
  4. 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs
  5. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses
  6. 2025A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Models
  7. 2025Rewrite to Jailbreak: Discover Learnable and Transferable Implicit Harmfulness Instruction
  8. 2024Improved Large Language Model Jailbreak Detection via Pretrained Embeddings