mlsanityData to 7 Oct 2026

"Moralized" Multi-Step Jailbreak Prompts: Black-Box Testing of Guardrails in Large Language Models for Verbal Attacks

arXiv 2411.16730 ↗First posted 23 Nov 2024cs.CR cs.AI cs.CLcited by 0

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

As the application of large language models continues to expand in various fields, it poses higher challenges to the effectiveness of identifying harmful content generation and guardrail mechanisms. This research aims to evaluate the guardrail effectiveness of GPT-4o, Grok-2 Beta, Llama 3.1 (405B), Gemini 1.5, and Claude 3.5 Sonnet through black-box testing of seemingly ethical multi-step jailbreak prompts. It conducts ethical attacks by designing an identical multi-step prompts that simulates the scenario of "corporate middle managers competing for promotions." The data results show that the guardrails of the above-mentioned LLMs were bypassed and the content of verbal attacks was generated. Claude 3.5 Sonnet's resistance to multi-step jailbreak prompts is more obvious. To ensure objectivity, the experimental process, black box test code, and enhanced guardrail code are uploaded to the GitHub repository: https://github.com/brucewang123456789/GeniusTrail.git.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 36 of its 62 references that are in the map. First posted on arXiv 23 Nov 2024.

Builds on

36 of 62 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Multi-Turn Context Jailbreak Attack on Large Language Models From First Principles8 Aug
  2. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey5 Jul
  3. 2024Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings19 Jun
  4. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models26 Mar
  5. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs8 Feb
  6. 2023Exploiting Novel GPT-4 APIs21 Dec
  7. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks16 Oct
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  9. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep

Other references · 27

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  7. 2021Ethical and social risks of harm from Language Modelscited by 957
  8. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  9. 2023A Comprehensive Overview of Large Language Modelscited by 639
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  11. 2024Large Language Models: A Surveycited by 506
  12. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
15 more
  1. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  2. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  3. 2022Predictability and Surprise in Large Generative Modelscited by 209
  4. 2023Evaluating and Mitigating Discrimination in Language Model Decisionscited by 117
  5. 2024Safeguarding Large Language Models: A Surveycited by 73
  6. 2024Building Guardrails for Large Language Modelscited by 61
  7. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  8. 2024Current state of LLM Risks and AI Guardrailscited by 44
  9. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  10. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  11. 2023Prompting Frameworks for Large Language Models: A Surveycited by 29
  12. 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
  13. 2024Benchmarking LLM Guardrails in Handling Multilingual Toxicitycited by 13
  14. 2024Mitigating Sycophancy in Decoder-Only Transformer Architectures: Synthetic Data Interventioncited by 5
  15. 2024Reducing Reasoning Costs: The Path of Optimization for Chain of Thought via Sparse Attention Mechanismcited by 4

Built on by

Not cited yet.