mlsanityData to 7 Oct 2026

All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks

arXiv 2401.09798 ↗First posted 18 Jan 2024cs.CL cs.AI cs.CYcited by 30

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMs), such as ChatGPT, encounter `jailbreak' challenges, wherein safeguards are circumvented to generate ethically harmful prompts. This study introduces a straightforward black-box method for efficiently crafting jailbreak prompts, addressing the significant complexity and computational costs associated with conventional methods. Our technique iteratively transforms harmful prompts into benign expressions directly utilizing the target LLM, predicated on the hypothesis that LLMs can autonomously generate expressions that evade safeguards. Through experiments conducted with ChatGPT (GPT-3.5 and GPT-4) and Gemini-Pro, our method consistently achieved an attack success rate exceeding 80% within an average of five iterations for forbidden questions and proved robust against model updates. The jailbreak prompts generated were not only naturally-worded and succinct but also challenging to defend against. These findings suggest that the creation of effective jailbreak prompts is less complex than previously believed, underscoring the heightened risk posed by black-box jailbreak attacks.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 15 of its 25 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

15 of 25 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks16 Oct
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  3. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  4. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  5. 2023Detecting Language Model Attacks with Perplexity27 Aug
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  7. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 8

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  6. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  7. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  8. 2023A Review of ChatGPT Applications in Education, Marketing, Software Engineering, and Healthcare: Benefits, Drawbacks, and Research Directionscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 12; 2026: 4.

30 papers.

Sort:

By area:

  1. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  2. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  3. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  4. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 98
  5. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  6. 2024PAL: Proxy-Guided Black-Box Attack on Large Language Modelscited by 46
  7. 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulationcited by 31
  8. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  9. 2024Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?cited by 15
  10. 2025One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMscited by 12
  11. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  12. 2024Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroringcited by 10
18 more
  1. 2024FLRT: Fluent Student-Teacher Redteamingcited by 9
  2. 2025LLMs Caught in the Crossfire: Malware Requests and Jailbreak Challengescited by 9
  3. 2025PiCo: Jailbreaking Multimodal Large Language Models via Pictorial Code Contextualizationcited by 8
  4. 2024LoRA-Guard: Parameter-Efficient Guardrail Adaptation for Content Moderation of Large Language Modelscited by 7
  5. 2025ASTRA: Autonomous Spatial-Temporal Red-teaming for AI Software Assistantscited by 6
  6. 2025TurboFuzzLLM: Turbocharging Mutation-based Fuzzing for Effectively Jailbreaking Large Language Models in Practicecited by 5
  7. 2025Imperceptible Jailbreaking against Large Language Modelscited by 3
  8. 2024CFSafety: Comprehensive Fine-grained Safety Assessment for LLMscited by 1
  9. 2024Red-Teaming Text-to-Image Models via In-Context Experience Replay and Semantic-Preserving Prompt Rewritingcited by 1
  10. 2025SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Modelscited by 1
  11. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  12. 2026Overlooked Safety Vulnerability in LLMs: Malicious Intelligent Optimization Algorithm Request and its Jailbreakcited by 1
  13. 2026TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language Modelscited by 1
  14. 2026Benchmarking the Safety of Large Language Models for Robotic Health Attendant Controlcited by 1
  15. 2025TestDG: Test-time Domain Generalization for Continual Test-time Adaptationcited by 0
  16. 2025Improving LLM Outputs Against Jailbreak Attacks with Expert Model Integrationcited by 0
  17. 2025ASRJam: Human-Friendly AI Speech Jamming to Prevent Automated Phone Scamscited by 0
  18. 2026Jailbreaking the Matrix: Nullspace Steering for Controlled Model Subversioncited by 0