mlsanityData to 14 Oct 2026

Toward Principled LLM Safety Testing: Solving the Jailbreak Oracle Problem

arXiv 2506.17299 ↗First posted 17 Jun 2025cs.CR cs.AI cs.LGcited by 4

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

As large language models (LLMs) become increasingly deployed in safety-critical applications, the lack of systematic methods to assess their vulnerability to jailbreak attacks presents a critical security gap. We introduce the jailbreak oracle problem: given a model, prompt, and decoding strategy, determine whether a jailbreak response can be generated with likelihood exceeding a specified threshold. This formalization enables a principled study of jailbreak vulnerabilities. Answering the jailbreak oracle problem poses significant computational challenges, as the search space grows exponentially with response length. We present Boa, the first system designed for efficiently solving the jailbreak oracle problem. Boa employs a two-phase search strategy: (1) breadth-first sampling to identify easily accessible jailbreaks, followed by (2) depth-first priority search guided by fine-grained safety scores to systematically explore promising yet low-probability paths. Boa enables rigorous security assessments including systematic defense evaluation, standardized comparison of red team attacks, and model certification under extreme adversarial conditions. Code is available at https://github.com/shuyilinn/BOA/tree/mlsys2026ae

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 41 of its 53 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

41 of 53 references.

Closest ancestors · 22

Same direction, newest first

  1. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming31 Jan
  2. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreaks13 Dec
  3. 2024Best-of-N Jailbreaking4 Dec
  4. 2024Rapid Response: Mitigating LLM Jailbreaks with a Few Examples12 Nov
  5. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs22 Jul
  6. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey5 Jul
  7. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models26 Jun
  8. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models13 Jun
  9. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs11 Apr
  10. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  11. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models28 Mar
  12. 2024Query-Based Adversarial Prompt Generation19 Feb
10 more
  1. 2024A StrongREJECT for Empty Jailbreaks15 Feb
  2. 2024Weak-to-Strong Jailbreaking on Large Language Models30 Jan
  3. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  5. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  6. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  7. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  9. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  10. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks24 May

Other references · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2025Gemma 3 Technical Reportcited by 1,824
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  9. 2018Hierarchical Neural Story Generationcited by 1,193
  10. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  11. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  12. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
7 more
  1. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  2. 20242 OLMo 2 Furiouscited by 340
  3. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  4. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 267
  5. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
  6. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 114
  7. 2025LLM-Safety Evaluations Lack Robustnesscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Quant Fever, Reasoning Blackholes, Schrodinger's Compliance, and More: Probing GPT-OSS-20Bcited by 1
  2. 2025LLM Reinforcement in Contextcited by 0
  3. 2025Invasive Context Engineering to Control Large Language Modelscited by 0
  4. 2026Toward a Principled Framework for Agent Safety Measurementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploring Jailbreak Attacks on LLMs through Intent Concealment and Diversion
  2. 2025Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval
  3. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks
  4. 2026The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
  5. 2026SoK: Robustness in Large Language Models against Jailbreak Attacks
  6. 2024Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models
  7. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding
  8. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks