# Best-of-N Jailbreaking

arXiv: https://arxiv.org/abs/2412.03556 · HTML: https://mlsanity.com/paper/2412.03556/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 44 of its 73 references that are in the map.
- First posted on arXiv 4 Dec 2024.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 30 of 44)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://mlsanity.com/paper/2310.04451/) (2023)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://mlsanity.com/paper/2401.06373/) (2024)
- [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://mlsanity.com/paper/2309.10253/) (2023)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://mlsanity.com/paper/2404.02151/) (2024)
- [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://mlsanity.com/paper/2310.06987/) (2023)
- [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://mlsanity.com/paper/2310.06387/) (2023)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts](https://mlsanity.com/paper/2311.05608/) (2023)
- [A StrongREJECT for Empty Jailbreaks](https://mlsanity.com/paper/2402.10260/) (2024)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [Improving Alignment and Robustness with Circuit Breakers](https://mlsanity.com/paper/2406.04313/) (2024)
- [Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation](https://mlsanity.com/paper/2311.03348/) (2023)
- [Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts](https://mlsanity.com/paper/2402.16822/) (2024)
- [How Robust is Google's Bard to Adversarial Image Attacks?](https://mlsanity.com/paper/2309.11751/) (2023)
- [Image Hijacks: Adversarial Images can Control Generative Models at Runtime](https://mlsanity.com/paper/2309.00236/) (2023)
- [Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models](https://mlsanity.com/paper/2403.09792/) (2024)
- [Open Sesame! Universal Black Box Jailbreaking of Large Language Models](https://mlsanity.com/paper/2309.01446/) (2023)
- [Evaluating Frontier Models for Dangerous Capabilities](https://mlsanity.com/paper/2403.13793/) (2024)

## Built on by (most cited first, 30 of 76)

- [Building Production-Ready Probes For Gemini](https://mlsanity.com/paper/2601.11516/) (2026)
- [Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens](https://mlsanity.com/paper/2505.13775/) (2025)
- [Answer Matching Outperforms Multiple Choice for Language Model Evaluation](https://mlsanity.com/paper/2507.02856/) (2025)
- [Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!](https://mlsanity.com/paper/2504.09762/) (2025)
- [InfAlign: Inference-aware language model alignment](https://mlsanity.com/paper/2412.19792/) (2024)
- [Multilingual and Multi-Accent Jailbreaking of Audio LLMs](https://mlsanity.com/paper/2504.01094/) (2025)
- [Adversarial Reasoning at Jailbreaking Time](https://mlsanity.com/paper/2502.01633/) (2025)
- [ToolTweak: An Attack on Tool Selection in LLM-based Agents](https://mlsanity.com/paper/2510.02554/) (2025)
- [Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models](https://mlsanity.com/paper/2505.15406/) (2025)
- [SoK: Evaluating Jailbreak Guardrails for Large Language Models](https://mlsanity.com/paper/2506.10597/) (2025)
- [LLM-Safety Evaluations Lack Robustness](https://mlsanity.com/paper/2503.02574/) (2025)
- [How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition](https://mlsanity.com/paper/2603.15714/) (2026)
- [Beyond Linear Probes: Dynamic Safety Monitoring for Language Models](https://mlsanity.com/paper/2509.26238/) (2025)
- [Chain-of-Thought Hijacking](https://mlsanity.com/paper/2510.26418/) (2025)
- [Position: Adversarial ML for LLMs Is Not Making Any Progress](https://mlsanity.com/paper/2502.02260/) (2025)
- [Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming](https://mlsanity.com/paper/2601.18076/) (2026)
- [Unveiling the Basin-Like Loss Landscape in Large Language Models](https://mlsanity.com/paper/2505.17646/) (2025)
- [Shaping capabilities with token-level data filtering](https://mlsanity.com/paper/2601.21571/) (2026)
- [MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agents](https://mlsanity.com/paper/2503.10809/) (2025)
- [Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility](https://mlsanity.com/paper/2507.11630/) (2025)
- [Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulness](https://mlsanity.com/paper/2603.06612/) (2026)
- [Mode-Conditioning Unlocks Superior Test-Time Scaling](https://mlsanity.com/paper/2512.01127/) (2025)
- [MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety](https://mlsanity.com/paper/2605.01687/) (2026)
- [Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track](https://mlsanity.com/paper/2506.19882/) (2025)
- [Whither symbols in the era of advanced neural networks?](https://mlsanity.com/paper/2508.05776/) (2025)
- [Diffusion LLMs are Natural Adversaries for any LLM](https://mlsanity.com/paper/2511.00203/) (2025)
- [AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research](https://mlsanity.com/paper/2511.04316/) (2025)
- [LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Seconds](https://mlsanity.com/paper/2412.05232/) (2024)
- [Toward Principled LLM Safety Testing: Solving the Jailbreak Oracle Problem](https://mlsanity.com/paper/2506.17299/) (2025)
- [Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variations](https://mlsanity.com/paper/2510.16893/) (2025)

## Abstract (arXiv)

We introduce Best-of-N (BoN) Jailbreaking, a simple black-box algorithm that jailbreaks frontier AI systems across modalities. BoN Jailbreaking works by repeatedly sampling variations of a prompt with a combination of augmentations - such as random shuffling or capitalization for textual prompts - until a harmful response is elicited. We find that BoN Jailbreaking achieves high attack success rates (ASRs) on closed-source language models, such as 89% on GPT-4o and 78% on Claude 3.5 Sonnet when sampling 10,000 augmented prompts. Further, it is similarly effective at circumventing state-of-the-art open-source defenses like circuit breakers. BoN also seamlessly extends to other modalities: it jailbreaks vision language models (VLMs) such as GPT-4o and audio language models (ALMs) like Gemini 1.5 Pro, using modality-specific augmentations. BoN reliably improves when we sample more augmented prompts. Across all modalities, ASR, as a function of the number of samples (N), empirically follows power-law-like behavior for many orders of magnitude. BoN Jailbreaking can also be composed with other black-box algorithms for even more effective attacks - combining BoN with an optimized prefix attack achieves up to a 35% increase in ASR. Overall, our work indicates that, despite their capability, language models are sensitive to seemingly innocuous changes to inputs, which attackers can exploit across modalities.