mlsanityData to 14 Oct 2026

Intention Analysis Makes LLMs A Good Jailbreak Defender

arXiv 2401.06561 ↗First posted 12 Jan 2024cs.CLcited by 74

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Aligning large language models (LLMs) with human values, particularly when facing complex and stealthy jailbreak attacks, presents a formidable challenge. Unfortunately, existing methods often overlook this intrinsic nature of jailbreaks, which limits their effectiveness in such complex scenarios. In this study, we present a simple yet highly effective defense strategy, i.e., Intention Analysis (mathbbIA). mathbbIA works by triggering LLMs' inherent self-correct and improve ability through a two-stage process: 1) analyzing the essential intention of the user input, and 2) providing final policy-aligned responses based on the first round conversation. Notably, mathbbIA is an inference-only method, thus could enhance LLM safety without compromising their helpfulness. Extensive experiments on varying jailbreak benchmarks across a wide range of LLMs show that mathbbIA could consistently and significantly reduce the harmfulness in responses (averagely -48.2% attack success rate). Encouragingly, with our mathbbIA, Vicuna-7B even outperforms GPT-3.5 regarding attack success rate. We empirically demonstrate that, to some extent, mathbbIA is robust to errors in generated intentions. Further analyses reveal the underlying principle of mathbbIA: suppressing LLM's tendency to follow jailbreak prompts, thereby enhancing safety.

Key facts. Cited by 74 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 65 references that are in the map. First posted on arXiv 12 Jan 2024.

Builds on

33 of 65 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  2. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  4. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  6. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  8. 2021Ethical and social risks of harm from Language Modelscited by 966
  9. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  11. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  12. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
15 more
  1. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  2. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  3. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  4. 2023Detecting Language Model Attacks with Perplexitycited by 431
  5. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
  6. 2023Pretraining Language Models with Human Preferencescited by 247
  7. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  8. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  9. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 173
  10. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 170
  11. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  12. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  13. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  14. 2021A Unified Generative Framework for Aspect-Based Sentiment Analysiscited by 97
  15. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysiscited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 31; 2026: 29.

74 papers.

Sort:

By area:

  1. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
  2. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  3. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 70
  4. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 69
  5. 2024Large Language Model Safety: A Holistic Surveycited by 59
  6. 2024Revisiting Demonstration Selection Strategies in In-Context Learningcited by 58
  7. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 57
  8. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 47
  9. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attackscited by 37
  10. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 34
  11. 2025Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreakingcited by 34
  12. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 31
38 more
  1. 2024Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Modelscited by 23
  2. 2025Silent Leaks: Implicit Knowledge Extraction Attack on RAG Systems through Benign Queriescited by 19
  3. 2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hackingcited by 18
  4. 2025Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulationcited by 14
  5. 2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compressioncited by 13
  6. 2025Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMscited by 11
  7. 2025Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMscited by 9
  8. 2025Why Not Act on What You Know? Unleashing Safety Potential of LLMs via Self-Aware Guard Enhancementcited by 8
  9. 2024Zer0-Jack: A Memory-efficient Gradient-based Jailbreaking Method for Black-box Multi-modal Large Language Modelscited by 7
  10. 2025Proactive defense against LLM Jailbreakcited by 7
  11. 2026Sparse Autoencoders are Capable LLM Jailbreak Mitigatorscited by 6
  12. 2024PrisonBreak: Jailbreaking Large Language Models with at Most Twenty-Five Targeted Bit-flipscited by 5
  13. 2025Smoke and Mirrors: Jailbreaking LLM-based Code Generation via Implicit Malicious Promptscited by 5
  14. 2025AMIA: Automatic Masking and Joint Intention Analysis Makes LVLMs Robust Jailbreak Defenderscited by 5
  15. 2025sudoLLM: On Multi-role Alignment of Language Modelscited by 4
  16. 2025Chain-of-Thought Driven Adversarial Scenario Extrapolation for Robust Language Modelscited by 4
  17. 2025Exploring the Vulnerability of the Content Moderation Guardrail in Large Language Models via Intent Manipulationcited by 4
  18. 2025ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Momentscited by 4
  19. 2025Mitigating Jailbreaks with Intent-Aware LLMscited by 4
  20. 2026Jailbreaking LLMs & VLMs: Mechanisms, Evaluation, and Unified Defensecited by 3
  21. 2026AISA: Awakening Intrinsic Safety Awareness in Large Language Models against Jailbreak Attackscited by 3
  22. 2024Look Before You Leap: Enhancing Attention and Vigilance Regarding Harmful Content with GuidelineLLMcited by 2
  23. 2025How Jailbreak Defenses Work and Ensemble? A Mechanistic Investigationcited by 2
  24. 2025Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attackscited by 2
  25. 2025Attention-Aware GNN-based Input Defense against Multi-Turn LLM Jailbreakcited by 2
  26. 2025CoCoTen: Detecting Adversarial Inputs to Large Language Models through Latent Space Features of Contextual Co-occurrence Tensorscited by 2
  27. 2026MemPot: Defending Against Memory Extraction Attack with Optimized Honeypotscited by 2
  28. 2026Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generationcited by 2
  29. 2026Fail-Closed Alignment for Large Language Modelscited by 2
  30. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  31. 2026Layered LLM Defenses as an Ensemble: Access Tiers, Inference Cost, and the Measured Failure Correlation Between Defense Layerscited by 2
  32. 2025GraphAttack: Exploiting Representational Blindspots in LLM Safety Mechanismscited by 1
  33. 2025Beyond Surface-Level Detection: Towards Cognitive-Driven Defense Against Jailbreak Attacks via Meta-Operations Reasoningcited by 1
  34. 2025Retrieval-Augmented Defense: Adaptive and Controllable Jailbreak Prevention for Large Language Modelscited by 1
  35. 2025Beyond Context: Large Language Models' Failure to Grasp Users' Intentcited by 1
  36. 2026The Shadow Self: Intrinsic Value Misalignment in Large Language Model Agentscited by 1
  37. 2026Spider-Sense: Intrinsic Risk Sensing for Efficient Agent Defense with Hierarchical Adaptive Screeningcited by 1
  38. 2026Proactive Rejection and Grounded Execution: A Dual-Stage Intent Analysis Paradigm for Safe and Efficient AIoT Smart Homescited by 1

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Driven Jailbreak
  2. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks
  3. 2025DETAM: Defending LLMs Against Jailbreak Attacks via Targeted Attention Modification
  4. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks
  5. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs
  6. 2025You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defense
  7. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily
  8. 2024Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis