mlsanityData to 7 Oct 2026

GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis

arXiv 2402.13494 ↗First posted 21 Feb 2024cs.CL cs.CRcited by 87

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMs) face threats from jailbreak prompts. Existing methods for detecting jailbreak prompts are primarily online moderation APIs or finetuned LLMs. These strategies, however, often require extensive and resource-intensive data collection and training processes. In this study, we propose GradSafe, which effectively detects jailbreak prompts by scrutinizing the gradients of safety-critical parameters in LLMs. Our method is grounded in a pivotal observation: the gradients of an LLM's loss for jailbreak prompts paired with compliance response exhibit similar patterns on certain safety-critical parameters. In contrast, safe prompts lead to different gradient patterns. Building on this observation, GradSafe analyzes the gradients from prompts (paired with compliance responses) to accurately detect jailbreak prompts. We show that GradSafe, applied to Llama-2 without further training, outperforms Llama Guard, despite its extensive finetuning with a large dataset, in detecting jailbreak prompts. This superior performance is consistent across both zero-shot and adaptation scenarios, as evidenced by our evaluations on ToxicChat and XSTest. The source code is available at https://github.com/xyq7/GradSafe.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 16 of its 30 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

16 of 30 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  8. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  9. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  10. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  11. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  12. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
3 more
  1. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  2. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 206
  3. 2021Ruddit: Norms of Offensiveness for English Reddit Commentscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 37; 2026: 38.

87 papers.

Sort:

By area:

  1. 2025Securing AI Agents with Information-Flow Controlcited by 108
  2. 2024SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steeringcited by 45
  3. 2025Improving LLM Safety Alignment with Dual-Objective Optimizationcited by 40
  4. 2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibrationcited by 30
  5. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden Statescited by 29
  6. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 28
  7. 2025Defending against Indirect Prompt Injection by Instruction Detectioncited by 25
  8. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignmentcited by 24
  9. 2024Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Modelscited by 23
  10. 2025Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitorscited by 21
  11. 2025SafeKey: Amplifying Aha-Moment Insights for Safety Reasoningcited by 20
  12. 2024MirrorCheck: Efficient Adversarial Defense for Vision-Language Modelscited by 19
38 more
  1. 2024VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wildcited by 17
  2. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
  3. 2025AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Modelscited by 16
  4. 2025Attack and defense techniques in large language models: A survey and new perspectivescited by 13
  5. 2025LLM Jailbreak Detection for (Almost) Free!cited by 13
  6. 2024SATA: A Paradigm for LLM Jailbreak via Simple Assistive Task Linkagecited by 10
  7. 2025Jailbreak Attack Initializations as Extractors of Compliance Directionscited by 10
  8. 2025UIPE: Enhancing LLM Unlearning by Removing Knowledge Related to Forgetting Targetscited by 10
  9. 2025JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Modelcited by 10
  10. 2025Detecting and Filtering Unsafe Training Data via Data Attribution with Denoised Representationcited by 9
  11. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  12. 2025Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debatecited by 8
  13. 2024GenTel-Safe: A Unified Benchmark and Shielding Framework for Defending Against Prompt Injection Attackscited by 7
  14. 2026SafeAgent: A Runtime Protection Architecture for Agentic Systemscited by 7
  15. 2024Refusing Safe Prompts for Multi-modal Large Language Modelscited by 6
  16. 2025DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Modelscited by 5
  17. 2026AI Psychosis: Does Conversational AI Amplify Delusion-Related Language?cited by 5
  18. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attackscited by 4
  19. 2025QGuard:Question-based Zero-shot Guard for Multi-modal LLM Safetycited by 4
  20. 2025SafeLLM: Unlearning Harmful Outputs from Large Language Models against Jailbreak Attackscited by 4
  21. 2026Defending Large Language Models Against Jailbreak Attacks via In-Decoding Safety-Awareness Probingcited by 4
  22. 2026Detecting and Suppressing Reward Hacking with Gradient Fingerprintscited by 4
  23. 2024PrisonBreak: Jailbreaking Large Language Models with at Most Twenty-Five Targeted Bit-flipscited by 3
  24. 2025MIST: Jailbreaking Black-box Large Language Models via Iterative Semantic Tuningcited by 3
  25. 2025When Good Sounds Go Adversarial: Jailbreaking Audio-Language Models with Benign Inputscited by 3
  26. 2025CCFC: Core & Core-Full-Core Dual-Track Defense for LLM Jailbreak Protectioncited by 3
  27. 2025Online Learning Defense against Iterative Jailbreak Attacks via Prompt Optimizationcited by 3
  28. 2026ALERT: Zero-shot LLM Jailbreak Detection via Internal Discrepancy Amplificationcited by 3
  29. 2026Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shiftcited by 3
  30. 2024A Framework for Real-time Safeguarding the Text Generation of Large Language Modelcited by 2
  31. 2025Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attackscited by 2
  32. 2025Attention-Aware GNN-based Input Defense against Multi-Turn LLM Jailbreakcited by 2
  33. 2025ALMGuard: Safety Shortcuts and Where to Find Them as Guardrails for Audio-Language Modelscited by 2
  34. 2026STAR-S: Improving Safety Alignment through Self-Taught Reasoning on Safety Rulescited by 2
  35. 2026Fail-Closed Alignment for Large Language Modelscited by 2
  36. 2026Unsafe by Reciprocity: How Generation-Understanding Coupling Undermines Safety in Unified Multimodal Modelscited by 2
  37. 2026CASA: Classification Augmented with Safety Attention for Robust Multimodal Alignmentcited by 2
  38. 2026An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Modelscited by 2

50 most cited of 87 All 87 (JSON)