# All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks

arXiv: https://arxiv.org/abs/2401.09798 · HTML: https://mlsanity.com/paper/2401.09798/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 18 Jan 2024.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://mlsanity.com/paper/2309.00614/) (2023)
- [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023)
- [SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2310.03684/) (2023)
- [Detecting Language Model Attacks with Perplexity](https://mlsanity.com/paper/2308.14132/) (2023)
- [Toxicity in ChatGPT: Analyzing Persona-assigned Language Models](https://mlsanity.com/paper/2304.05335/) (2023)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks](https://mlsanity.com/paper/2310.10844/) (2023)
- [Open Sesame! Universal Black Box Jailbreaking of Large Language Models](https://mlsanity.com/paper/2309.01446/) (2023)
- [A Review of ChatGPT Applications in Education, Marketing, Software Engineering, and Healthcare: Benefits, Drawbacks, and Research Directions](https://mlsanity.com/paper/2305.00237/) (2023)

## Built on by (most cited first, 30 of 32)

- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://mlsanity.com/paper/2404.02151/) (2024)
- [Jailbreak Attacks and Defenses Against Large Language Models: A Survey](https://mlsanity.com/paper/2407.04295/) (2024)
- [Improved Techniques for Optimization-Based Jailbreaking on Large Language Models](https://mlsanity.com/paper/2405.21018/) (2024)
- [Coercing LLMs to do and reveal (almost) anything](https://mlsanity.com/paper/2402.14020/) (2024)
- [PAL: Proxy-Guided Black-Box Attack on Large Language Models](https://mlsanity.com/paper/2402.09674/) (2024)
- [AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulation](https://mlsanity.com/paper/2410.09040/) (2024)
- [JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models](https://mlsanity.com/paper/2406.09321/) (2024)
- [Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?](https://mlsanity.com/paper/2412.03235/) (2024)
- [AutoJailbreak: Exploring Jailbreak Attacks and Defenses through a Dependency Lens](https://mlsanity.com/paper/2406.03805/) (2024)
- [Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring](https://mlsanity.com/paper/2410.21083/) (2024)
- [One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMs](https://mlsanity.com/paper/2505.17598/) (2025)
- [Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models](https://mlsanity.com/paper/2503.01742/) (2025)
- [FLRT: Fluent Student-Teacher Redteaming](https://mlsanity.com/paper/2407.17447/) (2024)
- [LLMs Caught in the Crossfire: Malware Requests and Jailbreak Challenges](https://mlsanity.com/paper/2506.10022/) (2025)
- [PiCo: Jailbreaking Multimodal Large Language Models via Pictorial Code Contextualization](https://mlsanity.com/paper/2504.01444/) (2025)
- [LoRA-Guard: Parameter-Efficient Guardrail Adaptation for Content Moderation of Large Language Models](https://mlsanity.com/paper/2407.02987/) (2024)
- [TurboFuzzLLM: Turbocharging Mutation-based Fuzzing for Effectively Jailbreaking Large Language Models in Practice](https://mlsanity.com/paper/2502.18504/) (2025)
- [ASTRA: Autonomous Spatial-Temporal Red-teaming for AI Software Assistants](https://mlsanity.com/paper/2508.03936/) (2025)
- [Imperceptible Jailbreaking against Large Language Models](https://mlsanity.com/paper/2510.05025/) (2025)
- [TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language Models](https://mlsanity.com/paper/2603.03081/) (2026)
- [CFSafety: Comprehensive Fine-grained Safety Assessment for LLMs](https://mlsanity.com/paper/2410.21695/) (2024)
- [Red-Teaming Text-to-Image Models via In-Context Experience Replay and Semantic-Preserving Prompt Rewriting](https://mlsanity.com/paper/2411.16769/) (2024)
- [SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Models](https://mlsanity.com/paper/2506.18543/) (2025)
- [Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation](https://mlsanity.com/paper/2508.10404/) (2025)
- [Overlooked Safety Vulnerability in LLMs: Malicious Intelligent Optimization Algorithm Request and its Jailbreak](https://mlsanity.com/paper/2601.00213/) (2026)
- [Benchmarking the Safety of Large Language Models for Robotic Health Attendant Control](https://mlsanity.com/paper/2604.26577/) (2026)
- [TestDG: Test-time Domain Generalization for Continual Test-time Adaptation](https://mlsanity.com/paper/2504.04981/) (2025)
- [Improving LLM Outputs Against Jailbreak Attacks with Expert Model Integration](https://mlsanity.com/paper/2505.17066/) (2025)
- [ASRJam: Human-Friendly AI Speech Jamming to Prevent Automated Phone Scams](https://mlsanity.com/paper/2506.11125/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs), such as ChatGPT, encounter `jailbreak' challenges, wherein safeguards are circumvented to generate ethically harmful prompts. This study introduces a straightforward black-box method for efficiently crafting jailbreak prompts, addressing the significant complexity and computational costs associated with conventional methods. Our technique iteratively transforms harmful prompts into benign expressions directly utilizing the target LLM, predicated on the hypothesis that LLMs can autonomously generate expressions that evade safeguards. Through experiments conducted with ChatGPT (GPT-3.5 and GPT-4) and Gemini-Pro, our method consistently achieved an attack success rate exceeding 80% within an average of five iterations for forbidden questions and proved robust against model updates. The jailbreak prompts generated were not only naturally-worded and succinct but also challenging to defend against. These findings suggest that the creation of effective jailbreak prompts is less complex than previously believed, underscoring the heightened risk posed by black-box jailbreak attacks.