mlsanityData to 14 Oct 2026

Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization

arXiv 2311.09096 ↗First posted 15 Nov 2023cs.CLcited by 170

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

While significant attention has been dedicated to exploiting weaknesses in LLMs through jailbreaking attacks, there remains a paucity of effort in defending against these attacks. We point out a pivotal factor contributing to the success of jailbreaks: the intrinsic conflict between the goals of being helpful and ensuring safety. Accordingly, we propose to integrate goal prioritization at both training and inference stages to counteract. Implementing goal prioritization during inference substantially diminishes the Attack Success Rate (ASR) of jailbreaking from 66.4% to 3.6% for ChatGPT. And integrating goal prioritization into model training reduces the ASR from 71.0% to 6.6% for Llama2-13B. Remarkably, even in scenarios where no jailbreaking samples are included during training, our approach slashes the ASR by half. Additionally, our findings reveal that while stronger LLMs face greater safety risks, they also possess a greater capacity to be steered towards defending against such attacks, both because of their stronger ability in instruction following. Our work thus contributes to the comprehension of jailbreaking attacks and defenses, and sheds light on the relationship between LLMs' capability and safety. Our code is available at urlhttps://github.com/thu-coai/JailbreakDefenseGoalPriority.

Key facts. Cited by 170 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 18 of its 31 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

18 of 31 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  2. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  3. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  4. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  6. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  7. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 11

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  6. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  7. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  8. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  9. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  10. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  11. 2023Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimationcited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 73; 2025: 65; 2026: 32.

170 papers.

Sort:

By area:

  1. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  2. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 247
  3. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 216
  4. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  5. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 156
  6. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 145
  7. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 131
  8. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  9. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  10. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  11. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  12. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 125
38 more
  1. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  2. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimizationcited by 118
  3. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  4. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 106
  5. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  6. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 96
  7. 2024Detoxifying Large Language Models via Knowledge Editingcited by 87
  8. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 86
  9. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 83
  10. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  11. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 80
  12. 2024Safeguarding Large Language Models: A Surveycited by 78
  13. 2024Intention Analysis Makes LLMs A Good Jailbreak Defendercited by 74
  14. 2024Defending LLMs against Jailbreaking Attacks via Backtranslationcited by 70
  15. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 70
  16. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 70
  17. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 69
  18. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 65
  19. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 64
  20. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 63
  21. 2025Foot-In-The-Door: A Multi-turn Jailbreak for LLMscited by 54
  22. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Railscited by 52
  23. 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledgecited by 51
  24. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  25. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  26. 2024Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspectivecited by 48
  27. 2024GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Modelscited by 48
  28. 2024When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Searchcited by 47
  29. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  30. 2024Distract Large Language Models for Automatic Jailbreak Attackcited by 45
  31. 2024Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characterscited by 42
  32. 2026AgentLAB: Benchmarking LLM Agents against Long-Horizon Attackscited by 39
  33. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 37
  34. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attackscited by 37
  35. 2024PARDEN, Can You Repeat That? Defending against Jailbreaks via Repetitioncited by 35
  36. 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 34
  37. 2025DoomArena: A framework for Testing AI Agents Against Evolving Security Threatscited by 32
  38. 2024Defending Jailbreak Prompts via In-Context Adversarial Gamecited by 29

50 most cited of 170 All 170 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Breaking the Ceiling: Exploring the Potential of Jailbreak Attacks through Expanding Strategy Space
  2. 2025SAID: Safety-Aware Intent Defense via Prefix Probing for Large Language Models
  3. 2026Minimal, Local, Causal Explanations for Jailbreak Success in Large Language Models
  4. 2024Harnessing Task Overload for Scalable Jailbreak Attacks on Large Language Models
  5. 2025Defending Large Language Models Against Jailbreak Exploits with Responsible AI Considerations
  6. 2024No Free Lunch for Defending Against Prefilling Attack by In-Context Learning
  7. 2024Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models
  8. 2026Defending Large Language Models Against Jailbreak Attacks via In-Decoding Safety-Awareness Probing