mlsanityData to 7 Oct 2026

Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes

arXiv 2403.00867 ↗First posted 1 Mar 2024cs.CR cs.AI cs.CLcited by 63

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMs) are becoming a prominent generative AI tool, where the user enters a query and the LLM generates an answer. To reduce harm and misuse, efforts have been made to align these LLMs to human values using advanced training techniques such as Reinforcement Learning from Human Feedback (RLHF). However, recent studies have highlighted the vulnerability of LLMs to adversarial jailbreak attempts aiming at subverting the embedded safety guardrails. To address this challenge, this paper defines and investigates the Refusal Loss of LLMs and then proposes a method called Gradient Cuff to detect jailbreak attempts. Gradient Cuff exploits the unique properties observed in the refusal loss landscape, including functional values and its smoothness, to design an effective two-step detection strategy. Experimental results on two aligned LLMs (LLaMA-2-7B-Chat and Vicuna-7B-V1.5) and six types of jailbreak attacks (GCG, AutoDAN, PAIR, TAP, Base64, and LRL) show that Gradient Cuff can significantly improve the LLM's rejection capability for malicious jailbreak queries, while maintaining the model's performance for benign user queries by adjusting the detection threshold.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 30 of its 35 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

30 of 35 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding14 Feb
  2. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  4. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  5. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  6. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  7. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  8. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  10. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 20

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  9. 2018Hierarchical Neural Story Generationcited by 1,182
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  11. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  12. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
8 more
  1. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  2. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  3. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
  4. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 160
  5. 2018Query-Efficient Hard-label Black-box Attack:An Optimization-based Approachcited by 130
  6. 2019A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimizationcited by 66
  7. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63
  8. 2022In conversation with Artificial Intelligence: aligning language models with human valuescited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 24; 2026: 22.

63 papers.

Sort:

By area:

  1. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 80
  4. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  5. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  6. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 67
  7. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  8. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  9. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 23
  10. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  11. 2025X-Boundary: Establishing Exact Safety Boundary to Shield LLMs from Multi-Turn Jailbreaks without Compromising Usabilitycited by 20
  12. 2024VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wildcited by 17
38 more
  1. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
  2. 2024LLMScan: Causal Scan for LLM Misbehavior Detectioncited by 13
  3. 2024Adversarial Representation Engineering: A General Model Editing Framework for Large Language Modelscited by 12
  4. 2025Think Twice, Generate Once: Safeguarding by Progressive Self-Reflectioncited by 11
  5. 2024Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitationscited by 10
  6. 2024PrimeGuard: Safe and Helpful LLMs through Tuning-Free Routingcited by 10
  7. 2025Jailbreak Attack Initializations as Extractors of Compliance Directionscited by 10
  8. 2025Self-Destructive Language Modelcited by 10
  9. 2024Decoupled Alignment for Robust Plug-and-Play Adaptationcited by 9
  10. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  11. 2026Knowledge-Driven Multi-Turn Jailbreaking on Large Language Modelscited by 6
  12. 2025CARE: Decoding Time Safety Alignment via Rollback and Introspection Interventioncited by 5
  13. 2025SafeLLM: Unlearning Harmful Outputs from Large Language Models against Jailbreak Attackscited by 4
  14. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  15. 2026Detecting and Suppressing Reward Hacking with Gradient Fingerprintscited by 4
  16. 2024PrisonBreak: Jailbreaking Large Language Models with at Most Twenty-Five Targeted Bit-flipscited by 3
  17. 2025STShield: Single-Token Sentinel for Real-Time Jailbreak Detection in Large Language Modelscited by 3
  18. 2025MIST: Jailbreaking Black-box Large Language Models via Iterative Semantic Tuningcited by 3
  19. 2025CCFC: Core & Core-Full-Core Dual-Track Defense for LLM Jailbreak Protectioncited by 3
  20. 2026ALERT: Zero-shot LLM Jailbreak Detection via Internal Discrepancy Amplificationcited by 3
  21. 2026Concept-Based Dictionary Learning for Inference-Time Safety in Vision Language Action Modelscited by 3
  22. 2025Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attackscited by 2
  23. 2025ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Momentscited by 2
  24. 2025JaiLIP: Jailbreaking Vision-Language Models via Loss Guided Image Perturbationcited by 2
  25. 2026Fail-Closed Alignment for Large Language Modelscited by 2
  26. 2026Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representationscited by 2
  27. 2026An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Modelscited by 2
  28. 2026Test-Time Safety Alignmentcited by 2
  29. 2025Geometry-Guided Adversarial Prompt Detection via Curvature and Local Intrinsic Dimensioncited by 1
  30. 2025LiteLMGuard: Seamless and Lightweight On-Device Prompt Filtering for Safeguarding Small Language Models against Quantization-induced Risks and Vulnerabilitiescited by 1
  31. 2025SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Modelscited by 1
  32. 2025Advancing Jailbreak Strategies: A Hybrid Approach to Exploiting LLM Vulnerabilities and Bypassing Modern Defensescited by 1
  33. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  34. 2025ASGuard: Activation-Scaling Guard to Mitigate Targeted Jailbreaking Attackcited by 1
  35. 2025Graph-Regularized Sparse Autoencoders for LLM Safety Steeringcited by 1
  36. 2026ShallowJail: Steering Jailbreaks against Large Language Modelscited by 1
  37. 2026Robust Harmful Features Under Jailbreak Attacks: Mechanistic Evidence from Attention Head Specialization in Large Language Modelscited by 1
  38. 2026Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directionscited by 1

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  2. 2026MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety
  3. 2026Jailbreaking Leaves a Trace: Understanding and Detecting Jailbreak Attacks from Internal Representations of Large Language Models
  4. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks
  5. 2024Defending against Jailbreak through Early Exit Generation of Large Language Models
  6. 2024Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundaries
  7. 2025Do Internal Layers of LLMs Reveal Patterns for Jailbreak Detection?
  8. 2025Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning