mlsanityData to 13 Oct 2026

A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily

arXiv 2311.08268 ↗First posted 14 Nov 2023cs.CLcited by 259

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMs), such as ChatGPT and GPT-4, are designed to provide useful and safe responses. However, adversarial prompts known as 'jailbreaks' can circumvent safeguards, leading LLMs to generate potentially harmful content. Exploring jailbreak prompts can help to better reveal the weaknesses of LLMs and further steer us to secure them. Unfortunately, existing jailbreak methods either suffer from intricate manual design or require optimization on other white-box models, which compromises either generalization or efficiency. In this paper, we generalize jailbreak prompt attacks into two aspects: (1) Prompt Rewriting and (2) Scenario Nesting. Based on this, we propose ReNeLLM, an automatic framework that leverages LLMs themselves to generate effective jailbreak prompts. Extensive experiments demonstrate that ReNeLLM significantly improves the attack success rate while greatly reducing the time cost compared to existing baselines. Our study also reveals the inadequacy of current defense methods in safeguarding LLMs. Finally, we analyze the failure of LLMs defense from the perspective of prompt execution priority, and propose corresponding defense strategies. We hope that our research can catalyze both the academic community and LLMs developers towards the provision of safer and more regulated LLMs. The code is available at https://github.com/NJUNLP/ReNeLLM.

Key facts. Cited by 259 papers in the map (data through 13 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 32 of its 54 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

32 of 54 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  2. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  3. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  5. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  6. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts7 Jun
  7. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,856
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,657
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,348
  5. 2017Deep reinforcement learning from human preferencescited by 3,871
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  7. 2020Learning to summarize from human feedbackcited by 2,575
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,139
  9. 2022Red Teaming Language Models with Language Modelscited by 1,014
  10. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 817
  11. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 647
  12. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 354
13 more
  1. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 298
  2. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  3. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 277
  4. 2023Pretraining Language Models with Human Preferencescited by 246
  5. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  6. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
  7. 2020Recipes for Safety in Open-domain Chatbotscited by 170
  8. 2023Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigationscited by 157
  9. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 88
  10. 2022Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Modelscited by 68
  11. 2023ControlLLM: Augment Language Models with Tools by Searching on Graphscited by 59
  12. 2023ChatCounselor: A Large Language Models for Mental Health Supportcited by 44
  13. 2023PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillationcited by 25

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 1; 2024: 61; 2025: 116; 2026: 81.

259 papers.

Sort:

By area:

  1. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 229
  2. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  3. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  4. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 114
  5. 2024Detoxifying Large Language Models via Knowledge Editingcited by 87
  6. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 85
  7. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  8. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 79
  9. 2024Safeguarding Large Language Models: A Surveycited by 76
  10. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 71
  11. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 71
  12. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 70
38 more
  1. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Cluescited by 66
  2. 2024EasyJailbreak: A Unified Framework for Jailbreaking Large Language Modelscited by 59
  3. 2024SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Typescited by 59
  4. 2024Learn Your Reference Model for Real Good Alignmentcited by 57
  5. 2025Foot-In-The-Door: A Multi-turn Jailbreak for LLMscited by 54
  6. 2024garak: A Framework for Security Probing Large Language Modelscited by 48
  7. 2024Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspectivecited by 47
  8. 2024When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Searchcited by 47
  9. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 47
  10. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  11. 2024Distract Large Language Models for Automatic Jailbreak Attackcited by 45
  12. 2025A Mousetrap: Fooling Large Reasoning Models for Jailbreak with Chain of Iterative Chaoscited by 42
  13. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 41
  14. 2025AdaSteer: Your Aligned LLM is Inherently an Adaptive Jailbreak Defendercited by 38
  15. 2025AgentSafe: Safeguarding Large Language Model-based Multi-agent Systems via Hierarchical Data Managementcited by 36
  16. 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructionscited by 36
  17. 2024Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understandingcited by 34
  18. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 34
  19. 2025Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMscited by 34
  20. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 33
  21. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 31
  22. 2025SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearningcited by 31
  23. 2024Protecting Your LLMs with Information Bottleneckcited by 29
  24. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  25. 2024Foot In The Door: Understanding Large Language Model Jailbreaking via Cognitive Psychologycited by 26
  26. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 25
  27. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 24
  28. 2024HSF: Defending against Jailbreak Attacks with Hidden State Filteringcited by 23
  29. 2024Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Modelscited by 23
  30. 2024Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detectioncited by 22
  31. 2024JailbreakLens: Interpreting Jailbreak Mechanism in the Lens of Representation and Circuitcited by 22
  32. 2025Understanding and Enhancing the Transferability of Jailbreaking Attackscited by 22
  33. 2026SkillTrojan: Backdoor Attacks on Skill-Based Agent Systemscited by 22
  34. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 21
  35. 2024The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Modelscited by 20
  36. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 20
  37. 2024Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Coursecited by 19
  38. 2024SequentialBreak: Large Language Models Can be Fooled by Embedding Jailbreak Prompts into Sequential Prompt Chainscited by 19

50 most cited of 259 All 259 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models
  2. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  3. 2026The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
  4. 2025Jailbreaking with Universal Multi-Prompts
  5. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models
  6. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models
  7. 2024AutoBreach: Universal and Adaptive Jailbreaking with Efficient Wordplay-Guided Optimization
  8. 2025Advancing Jailbreak Strategies: A Hybrid Approach to Exploiting LLM Vulnerabilities and Bypassing Modern Defenses