mlsanityData to 7 Oct 2026

DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers

arXiv 2402.16914 ↗First posted 25 Feb 2024cs.CR cs.AI cs.CLcited by 91

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

The safety alignment of Large Language Models (LLMs) is vulnerable to both manual and automated jailbreak attacks, which adversarially trigger LLMs to output harmful content. However, current methods for jailbreaking LLMs, which nest entire harmful prompts, are not effective at concealing malicious intent and can be easily identified and rejected by well-aligned LLMs. This paper discovers that decomposing a malicious prompt into separated sub-prompts can effectively obscure its underlying malicious intent by presenting it in a fragmented, less detectable form, thereby addressing these limitations. We introduce an automatic prompt Decomposition and Reconstruction framework for jailbreak Attack (DrAttack). DrAttack includes three key components: (a) `Decomposition' of the original prompt into sub-prompts, (b) `Reconstruction' of these sub-prompts implicitly by in-context learning with semantically similar but harmless reassembling demo, and (c) a `Synonym Search' of sub-prompts, aiming to find sub-prompts' synonyms that maintain the original intent while jailbreaking LLMs. An extensive empirical study across multiple open-source and closed-source LLMs demonstrates that, with a significantly reduced number of queries, DrAttack obtains a substantial gain of success rate over prior SOTA prompt-only attackers. Notably, the success rate of 78.0% on GPT-4 with merely 15 queries surpassed previous art by 33.1%. The project is available at https://github.com/xirui-li/DrAttack.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 33 references that are in the map. First posted on arXiv 25 Feb 2024.

Builds on

18 of 33 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  2. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  5. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  6. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  7. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  8. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  9. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 155
  10. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  11. 2022Successive Prompting for Decomposing Complex Questionscited by 94
  12. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
2 more
  1. 2023LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large Language Modelcited by 21
  2. 2022Task-specific Pre-training and Prompt Decomposition for Knowledge Graph Population with Language Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 42; 2026: 25.

91 papers.

Sort:

By area:

  1. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  2. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  3. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  4. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimizationcited by 112
  5. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 94
  6. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  7. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  8. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60
  9. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  10. 2024Multi-Turn Context Jailbreak Attack on Large Language Models From First Principlescited by 54
  11. 2024Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characterscited by 39
  12. 2024RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agentcited by 39
38 more
  1. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 32
  2. 2024WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Responsecited by 28
  3. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  4. 2024Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundariescited by 26
  5. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  6. 2025Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitorscited by 21
  7. 2025Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuningcited by 19
  8. 2025The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaborationcited by 19
  9. 2024Breach By A Thousand Leaks: Unsafe Information Leakage in `Safe' AI Responsescited by 18
  10. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMscited by 18
  11. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
  12. 2024SequentialBreak: Large Language Models Can be Fooled by Embedding Jailbreak Prompts into Sequential Prompt Chainscited by 16
  13. 2024Jigsaw Puzzles: Splitting Harmful Questions to Jailbreak Large Language Modelscited by 15
  14. 2025Insights from Verification: Training a Verilog Generation LLM with Reinforcement Learning with Testbench Feedbackcited by 14
  15. 2025Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Modelscited by 13
  16. 2025LLM Jailbreak Detection for (Almost) Free!cited by 13
  17. 2025Multi-lingual Multi-turn Automated Red Teaming for LLMscited by 12
  18. 2025One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMscited by 12
  19. 2025Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMscited by 10
  20. 2025GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methodscited by 9
  21. 2025SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinationscited by 9
  22. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  23. 2024Can LLMs Deeply Detect Complex Malicious Queries? A Framework for Jailbreaking via Obfuscating Intentcited by 8
  24. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  25. 2025Jailbreaking to Jailbreakcited by 8
  26. 2025Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMscited by 8
  27. 2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
  28. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
  29. 2025From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMscited by 6
  30. 2025ASTRA: Autonomous Spatial-Temporal Red-teaming for AI Software Assistantscited by 6
  31. 2025CluCERT: Certifying LLM Robustness via Clustering-Guided Denoising Smoothingcited by 6
  32. 2026What Matters For Safety Alignment?cited by 6
  33. 2025Bleeding Pathways: Vanishing Discriminability in LLM Hidden States Fuels Jailbreak Attackscited by 5
  34. 2025NeuRel-Attack: Neuron Relearning for Safety Disalignment in Large Language Modelscited by 5
  35. 2025When Memory Becomes a Vulnerability: Towards Multi-turn Jailbreak Attacks against Text-to-Image Generation Systemscited by 4
  36. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  37. 2024Preventing Jailbreak Prompts as Malicious Tools for Cybercriminals: A Cyber Defense Perspectivecited by 3
  38. 2024Prompt-based Unifying Inference Attack on Graph Neural Networkscited by 3

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
  2. 2026Break Me If You Can: Self-Jailbreaking of Aligned LLMs via Lexical Insertion Prompting
  3. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks
  4. 2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compression
  5. 2024Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings
  6. 2025HAMSA: Hijacking Aligned Compact Models via Stealthy Automation
  7. 2025Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning
  8. 2024AutoBreach: Universal and Adaptive Jailbreaking with Efficient Wordplay-Guided Optimization