mlsanityData to 7 Oct 2026

Best-of-N Jailbreaking

arXiv 2412.03556 ↗First posted 4 Dec 2024cs.CL cs.AI cs.LGcited by 71

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

We introduce Best-of-N (BoN) Jailbreaking, a simple black-box algorithm that jailbreaks frontier AI systems across modalities. BoN Jailbreaking works by repeatedly sampling variations of a prompt with a combination of augmentations - such as random shuffling or capitalization for textual prompts - until a harmful response is elicited. We find that BoN Jailbreaking achieves high attack success rates (ASRs) on closed-source language models, such as 89% on GPT-4o and 78% on Claude 3.5 Sonnet when sampling 10,000 augmented prompts. Further, it is similarly effective at circumventing state-of-the-art open-source defenses like circuit breakers. BoN also seamlessly extends to other modalities: it jailbreaks vision language models (VLMs) such as GPT-4o and audio language models (ALMs) like Gemini 1.5 Pro, using modality-specific augmentations. BoN reliably improves when we sample more augmented prompts. Across all modalities, ASR, as a function of the number of samples (N), empirically follows power-law-like behavior for many orders of magnitude. BoN Jailbreaking can also be composed with other black-box algorithms for even more effective attacks - combining BoN with an optimized prefix attack achieves up to a 35% increase in ASR. Overall, our work indicates that, despite their capability, language models are sensitive to seemingly innocuous changes to inputs, which attackers can exploit across modalities.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 44 of its 73 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

44 of 73 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Endless Jailbreaks with Bijection Learning2 Oct
  2. 2024h4rm3l: A language for Composable Jailbreak Attack Synthesis9 Aug
  3. 2024Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?4 Apr
  4. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  5. 2024A StrongREJECT for Empty Jailbreaks15 Feb
  6. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  7. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  8. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  10. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  11. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  12. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
3 more
  1. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  2. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 29

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  6. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  8. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  9. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  10. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  11. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  12. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
17 more
  1. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  2. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  3. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 177
  4. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
  5. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  6. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Modelscited by 121
  7. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 121
  8. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 99
  9. 2020Data Augmenting Contrastive Learning of Speech Representations in the Time Domaincited by 63
  10. 2023LLaSM: Large Language and Speech Modelcited by 56
  11. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  12. 2023Misusing Tools in Large Language Models With Visual Adversarial Examplescited by 40
  13. 2024Voice Jailbreak Attacks Against GPT-4ocited by 39
  14. 2024Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Modelscited by 31
  15. 2024Failures to Find Transferable Image Jailbreaks Between Vision-Language Modelscited by 27
  16. 2024Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systemscited by 24
  17. 2024Gradient-based Jailbreak Images for Multimodal Fusion Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 30; 2026: 39.

71 papers.

Sort:

By area:

  1. 2025Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokenscited by 41
  2. 2026Building Production-Ready Probes For Geminicited by 38
  3. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 37
  4. 2025Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!cited by 36
  5. 2024InfAlign: Inference-aware language model alignmentcited by 34
  6. 2025Adversarial Reasoning at Jailbreaking Timecited by 27
  7. 2025Multilingual and Multi-Accent Jailbreaking of Audio LLMscited by 25
  8. 2025ToolTweak: An Attack on Tool Selection in LLM-based Agentscited by 22
  9. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
  10. 2025LLM-Safety Evaluations Lack Robustnesscited by 16
  11. 2026How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competitioncited by 16
  12. 2025Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Modelscited by 15
38 more
  1. 2026Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teamingcited by 15
  2. 2025Position: Adversarial ML for LLMs Is Not Making Any Progresscited by 14
  3. 2025Beyond Linear Probes: Dynamic Safety Monitoring for Language Modelscited by 14
  4. 2025Unveiling the Basin-Like Loss Landscape in Large Language Modelscited by 13
  5. 2025Chain-of-Thought Hijackingcited by 13
  6. 2025MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agentscited by 12
  7. 2026Shaping capabilities with token-level data filteringcited by 11
  8. 2025Mode-Conditioning Unlocks Superior Test-Time Scalingcited by 8
  9. 2026Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulnesscited by 8
  10. 2025Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Trackcited by 7
  11. 2025Diffusion LLMs are Natural Adversaries for any LLMcited by 7
  12. 2026MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safetycited by 7
  13. 2025Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibilitycited by 6
  14. 2025Whither symbols in the era of advanced neural networks?cited by 6
  15. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Researchcited by 5
  16. 2025Toward Principled LLM Safety Testing: Solving the Jailbreak Oracle Problemcited by 4
  17. 2025Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variationscited by 4
  18. 2026Toward Universal and Transferable Jailbreak Attacks on Vision-Language Modelscited by 4
  19. 2026Visual Exclusivity Attacks: Automatic Multimodal Red Teaming via Agentic Planningcited by 4
  20. 2024LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Secondscited by 3
  21. 2026Jailbreaking Vision-Language Models Through the Visual Modalitycited by 3
  22. 2026Exposing LLM Safety Gaps Through Mathematical Encoding:New Attacks and Systematic Analysiscited by 3
  23. 2026Do Thinking Tokens Help with Safety?cited by 3
  24. 2025Phonetic Perturbations Reveal Tokenizer-Rooted Safety Gaps in LLMscited by 2
  25. 2025ConceptGuard: Neuro-Symbolic Safety Guardrails via Sparse Interpretable Jailbreak Conceptscited by 2
  26. 2025PLAGUE: Plug-and-play framework for Lifelong Adaptive Generation of Multi-turn Exploitscited by 2
  27. 2026Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossovercited by 2
  28. 2026Hijacking Large Audio-Language Models via Context-Agnostic and Imperceptible Auditory Prompt Injectioncited by 2
  29. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 2
  30. 2025A Generative Approach to LLM Harmfulness Mitigation with Red Flag Tokenscited by 1
  31. 2025Investigating Vulnerabilities and Defenses Against Audio-Visual Attacks: A Comprehensive Survey Emphasizing Multimodal Modelscited by 1
  32. 2025Multi-Agent Framework for Threat Mitigation and Resilience in AI-Based Systemscited by 1
  33. 2025Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language Modelscited by 1
  34. 2026Perturbation Probing: A Two-Pass-per-Prompt Diagnostic for FFN Behavioral Circuits in Aligned LLMscited by 1
  35. 2026Security of OpenClaw Agents: Fundamentals, Attacks, and Countermeasurescited by 1
  36. 2026Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attackscited by 1
  37. 2026FAPO: Fully Automated Prompt Optimization of Multi-Step LLM Pipelinescited by 1
  38. 2026Securing the AI Agent: A Unified Framework for Multi-Layer Agent Red Teamingcited by 1

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Jailbreak Mimicry: Automated Discovery of Narrative-Based Jailbreaks for Large Language Models
  2. 2024AutoBreach: Universal and Adaptive Jailbreaking with Efficient Wordplay-Guided Optimization
  3. 2026Jailbreaking Leaves a Trace: Understanding and Detecting Jailbreak Attacks from Internal Representations of Large Language Models
  4. 2025Defense-to-Attack: Bypassing Weak Defenses Enables Stronger Jailbreaks in Vision-Language Models
  5. 2026The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
  6. 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs
  7. 2026SAFEGuard: Detect Optimization-Based Jailbreak Attacks Through Harmful Semantic Analysis and Fluency Measurement
  8. 2024Defending against Jailbreak through Early Exit Generation of Large Language Models