mlsanityData to 7 Oct 2026

Jailbreaking Black Box Large Language Models in Twenty Queries

arXiv 2310.08419 ↗First posted 12 Oct 2023cs.LG cs.AIcited by 1,326

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

There is growing interest in ensuring that large language models (LLMs) align with human values. However, the alignment of such models is vulnerable to adversarial jailbreaks, which coax LLMs into overriding their safety guardrails. The identification of these vulnerabilities is therefore instrumental in understanding inherent weaknesses and preventing future misuse. To this end, we propose Prompt Automatic Iterative Refinement (PAIR), an algorithm that generates semantic jailbreaks with only black-box access to an LLM. PAIR -- which is inspired by social engineering attacks -- uses an attacker LLM to automatically generate jailbreaks for a separate targeted LLM without human intervention. In this way, the attacker LLM iteratively queries the target LLM to update and refine a candidate jailbreak. Empirically, PAIR often requires fewer than twenty queries to produce a jailbreak, which is orders of magnitude more efficient than existing algorithms. PAIR also achieves competitive jailbreaking success rates and transferability on open and closed-source LLMs, including GPT-3.5/4, Vicuna, and Gemini.

Key facts. Cited by 1,326 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 46 of its 69 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

46 of 69 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  2. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  3. 2023Detecting Language Model Attacks with Perplexity27 Aug
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  5. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots16 Jul
  6. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  9. 2013Intriguing properties of neural networkscited by 5,114
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,105
28 more
  1. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  2. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  3. 2023PaLM 2 Technical Reportcited by 1,154
  4. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  5. 2022Red Teaming Language Models with Language Modelscited by 983
  6. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  7. 2023BloombergGPT: A Large Language Model for Financecited by 923
  8. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  9. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  10. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  11. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  12. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 530
  13. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  14. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
  15. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 462
  16. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  17. 2019Provably Robust Deep Learning via Adversarially Trained Smoothed Classifierscited by 328
  18. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  19. 2023Are aligned neural networks adversarially aligned?cited by 305
  20. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  21. 2023Pretraining Language Models with Human Preferencescited by 245
  22. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  23. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  24. 2018Adversarial Attacks and Defences Competitioncited by 96
  25. 2022Parallel Rectangle Flip Attack: A Query-based Black-box Attack against Object Detectioncited by 79
  26. 2021Improving Question Answering Model Robustness with Synthetic Adversarial Data Generationcited by 77
  27. 2023Black Box Adversarial Prompting for Foundation Modelscited by 62
  28. 2021Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistantscited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 368; 2025: 477; 2026: 462.

1,326 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2024OpenAI o1 System Cardcited by 1,220
  3. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  4. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  5. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  6. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  7. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  8. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  9. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  10. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  11. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  12. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
38 more
  1. 2024A StrongREJECT for Empty Jailbreakscited by 352
  2. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  3. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  4. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  5. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  6. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  7. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  8. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  9. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  10. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  11. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  12. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  13. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  14. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  15. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  16. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  17. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  18. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  19. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  20. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  21. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  22. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  23. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  24. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  25. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 126
  26. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  27. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  28. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  29. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  30. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  31. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Modelscited by 121
  32. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  33. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  34. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  35. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  36. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 112
  37. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
  38. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110

50 most cited of 1,326 All 1,326 (JSON)